2023-10-30T20:00:03.628 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-30T20:00:03.629 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} --archive /home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403 --verbose -- /home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403/orig.config.yaml 2023-10-30T20:00:03.670 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T20:00:03.746 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7441403' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: cc36dd85d142c6fdb77515cad8af108cec98a86d valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: debuginfo: true flavor: default sha1: cc36dd85d142c6fdb77515cad8af108cec98a86d thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: d0db9f2f95b9d6344950185071cace2a50d296eb owner: scheduled_amathuri@teuthology priority: 90 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 1876 sha1: cc36dd85d142c6fdb77515cad8af108cec98a86d sleep_before_teardown: 0 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d0db9f2f95b9d6344950185071cace2a50d296eb targets: smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPYOadPJGZ6Dgmh4cUhlkNoyy6TxoaN3szWWY15zj2N5X3SpqTC5b49HSfaNZti1N2YwzFEGieaf0YsyG3qHro= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBDn5rofjGLsweyWrhB4yDSdrLYw67ggLPiGlfBteC8yQzlu+mrd5uMp5UD/byLTDpd2QtVo9ZCEH6z8FMVeIQk= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-30_17:23:32 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-30T20:00:03.746 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa; will attempt to use it 2023-10-30T20:00:03.747 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks 2023-10-30T20:00:03.748 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-30T20:00:03.750 INFO:teuthology.task.internal:Checking packages... 2023-10-30T20:00:03.774 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'cc36dd85d142c6fdb77515cad8af108cec98a86d' 2023-10-30T20:00:03.775 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-30T20:00:03.775 INFO:teuthology.packaging:ref: None 2023-10-30T20:00:03.775 INFO:teuthology.packaging:tag: None 2023-10-30T20:00:03.776 INFO:teuthology.packaging:branch: main 2023-10-30T20:00:03.776 INFO:teuthology.packaging:sha1: cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T20:00:03.776 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2023-10-30T20:00:04.014 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6877.g60d56b08 2023-10-30T20:00:04.015 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-30T20:00:04.022 INFO:teuthology.task.internal:no buildpackages task found 2023-10-30T20:00:04.022 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-30T20:00:04.027 INFO:teuthology.task.internal:Saving configuration 2023-10-30T20:00:04.041 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-30T20:00:04.047 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T20:00:04.073 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 19:55:06.895013', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPYOadPJGZ6Dgmh4cUhlkNoyy6TxoaN3szWWY15zj2N5X3SpqTC5b49HSfaNZti1N2YwzFEGieaf0YsyG3qHro='} 2023-10-30T20:00:04.098 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 19:55:06.893325', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBDn5rofjGLsweyWrhB4yDSdrLYw67ggLPiGlfBteC8yQzlu+mrd5uMp5UD/byLTDpd2QtVo9ZCEH6z8FMVeIQk='} 2023-10-30T20:00:04.098 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-30T20:00:04.104 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-10-30T20:00:04.105 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-10-30T20:00:04.105 INFO:teuthology.run_tasks:Running task console_log... 2023-10-30T20:00:04.177 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4b51213a60>, signals=[15]) 2023-10-30T20:00:04.178 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-30T20:00:04.183 INFO:teuthology.task.internal:Opening connections... 2023-10-30T20:00:04.184 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2023-10-30T20:00:04.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T20:00:04.268 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2023-10-30T20:00:04.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T20:00:04.352 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-30T20:00:04.359 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2023-10-30T20:00:04.377 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2023-10-30T20:00:04.378 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2023-10-30T20:00:04.434 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="CentOS Stream" 2023-10-30T20:00:04.434 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="9" 2023-10-30T20:00:04.434 INFO:teuthology.orchestra.run.smithi069.stdout:ID="centos" 2023-10-30T20:00:04.435 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="rhel fedora" 2023-10-30T20:00:04.435 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="9" 2023-10-30T20:00:04.435 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el9" 2023-10-30T20:00:04.435 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-30T20:00:04.435 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2023-10-30T20:00:04.436 INFO:teuthology.orchestra.run.smithi069.stdout:LOGO="fedora-logo-icon" 2023-10-30T20:00:04.436 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-30T20:00:04.436 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://centos.org/" 2023-10-30T20:00:04.436 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-30T20:00:04.436 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-30T20:00:04.437 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-30T20:00:04.437 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2023-10-30T20:00:04.462 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2023-10-30T20:00:04.480 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2023-10-30T20:00:04.481 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2023-10-30T20:00:04.537 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2023-10-30T20:00:04.537 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2023-10-30T20:00:04.537 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2023-10-30T20:00:04.538 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2023-10-30T20:00:04.538 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2023-10-30T20:00:04.538 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2023-10-30T20:00:04.538 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-30T20:00:04.538 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2023-10-30T20:00:04.539 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2023-10-30T20:00:04.539 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-30T20:00:04.539 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2023-10-30T20:00:04.539 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-30T20:00:04.539 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-30T20:00:04.540 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-30T20:00:04.540 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2023-10-30T20:00:04.562 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-30T20:00:04.571 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-30T20:00:04.576 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-30T20:00:04.576 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2023-10-30T20:00:04.579 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2023-10-30T20:00:04.595 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-30T20:00:04.601 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-30T20:00:04.601 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2023-10-30T20:00:04.636 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2023-10-30T20:00:04.667 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-30T20:00:04.719 INFO:teuthology.run_tasks:Running task kernel... 2023-10-30T20:00:04.733 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-30T20:00:04.734 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-30T20:00:04.735 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T20:00:04.736 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T20:00:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T20:00:04.752 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2023-10-30T20:00:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T20:00:04.753 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2023-10-30T20:00:04.808 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-361.el9.x86_64 2023-10-30T20:00:04.809 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.14.0-361.el9.x86_64 2023-10-30T20:00:04.809 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2023-10-30T20:00:04.810 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-361.el9.x86_64 2023-10-30T20:00:04.811 INFO:teuthology.task.kernel:Running kernel on smithi069: 5.14.0-361.el9.x86_64 2023-10-30T20:00:04.811 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum install -y kernel 2023-10-30T20:00:06.883 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 14 MB/s | 6.3 MB 00:00 2023-10-30T20:00:06.941 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 12 MB/s | 6.3 MB 00:00 2023-10-30T20:00:08.216 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2023-10-30T20:00:08.543 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2023-10-30T20:00:10.982 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 7.9 MB 00:02 2023-10-30T20:00:11.098 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 7.9 MB 00:02 2023-10-30T20:00:12.833 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 22 MB/s | 18 MB 00:00 2023-10-30T20:00:14.424 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 7.5 MB/s | 18 MB 00:02 2023-10-30T20:00:18.166 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.8 MB 00:00 2023-10-30T20:00:18.786 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 5.8 MB 00:02 2023-10-30T20:00:19.893 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 15 kB 00:00 2023-10-30T20:00:20.302 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2023-10-30T20:00:20.852 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 23 MB/s | 19 MB 00:00 2023-10-30T20:00:21.253 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 23 MB/s | 19 MB 00:00 2023-10-30T20:00:25.426 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2023-10-30T20:00:25.894 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2023-10-30T20:00:27.517 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-30T20:00:27.517 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-30T20:00:27.613 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:00:27.624 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:00:27.624 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2023-10-30T20:00:27.624 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:00:27.625 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:00:27.625 INFO:teuthology.orchestra.run.smithi069.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-10-30T20:00:27.625 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-10-30T20:00:27.625 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-10-30T20:00:27.626 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-10-30T20:00:27.626 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-10-30T20:00:27.626 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:00:27.626 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:00:27.626 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:00:27.627 INFO:teuthology.orchestra.run.smithi069.stdout:Install 4 Packages 2023-10-30T20:00:27.627 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:00:27.627 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 95 M 2023-10-30T20:00:27.628 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 120 M 2023-10-30T20:00:27.628 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:00:27.959 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-30T20:00:27.959 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-30T20:00:28.052 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:00:28.062 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:00:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2023-10-30T20:00:28.062 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:00:28.063 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:00:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-10-30T20:00:28.063 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2023-10-30T20:00:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-10-30T20:00:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-10-30T20:00:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-10-30T20:00:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:00:28.064 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:00:28.065 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:00:28.065 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2023-10-30T20:00:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:00:28.066 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 95 M 2023-10-30T20:00:28.066 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 120 M 2023-10-30T20:00:28.066 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:00:29.936 INFO:teuthology.orchestra.run.smithi069.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 2.4 MB/s | 5.3 MB 00:02 2023-10-30T20:00:32.378 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 1.3 MB/s | 5.3 MB 00:04 2023-10-30T20:00:35.353 INFO:teuthology.orchestra.run.smithi069.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 2.6 MB/s | 20 MB 00:07 2023-10-30T20:00:36.969 INFO:teuthology.orchestra.run.smithi069.stdout:(3/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 4.0 MB/s | 38 MB 00:09 2023-10-30T20:00:41.594 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): kernel-modules-core-5.14.0-378.el9.x86_6 3.5 MB/s | 32 MB 00:09 2023-10-30T20:00:42.411 INFO:teuthology.orchestra.run.smithi069.stdout:(4/4): kernel-modules-core-5.14.0-378.el9.x86_6 2.6 MB/s | 32 MB 00:12 2023-10-30T20:00:42.411 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:00:42.411 INFO:teuthology.orchestra.run.smithi069.stdout:Total 6.4 MB/s | 95 MB 00:14 2023-10-30T20:00:43.258 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:00:43.622 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:00:43.623 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:00:44.912 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:00:44.913 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:00:45.718 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): kernel-core-5.14.0-378.el9.x86_64.rpm 1.1 MB/s | 20 MB 00:17 2023-10-30T20:00:47.534 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:00:47.910 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 1.9 MB/s | 38 MB 00:19 2023-10-30T20:00:47.910 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:00:47.911 INFO:teuthology.orchestra.run.smithi183.stdout:Total 4.8 MB/s | 95 MB 00:19 2023-10-30T20:00:48.184 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-10-30T20:00:48.302 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T20:00:48.752 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:00:49.106 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:00:49.106 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:00:50.409 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:00:50.410 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:00:51.069 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T20:00:51.261 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T20:00:53.034 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:00:53.662 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-10-30T20:00:53.793 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T20:00:55.830 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T20:00:55.950 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T20:00:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T20:00:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T20:01:00.603 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T20:01:01.434 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T20:01:01.545 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T20:01:06.196 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T20:01:29.535 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T20:01:29.577 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-10-30T20:01:30.005 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T20:01:30.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-10-30T20:01:30.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T20:01:30.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T20:01:30.450 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T20:01:30.450 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:01:30.450 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:01:30.450 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-5.14.0-378.el9.x86_64 2023-10-30T20:01:30.451 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-10-30T20:01:30.451 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-10-30T20:01:30.452 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-10-30T20:01:30.452 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:01:30.452 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:01:30.935 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T20:01:31.083 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T20:01:31.084 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T20:01:31.084 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-10-30T20:01:31.085 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-30T20:01:31.085 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-30T20:01:31.085 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-30T20:01:31.085 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2023-10-30T20:01:31.085 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T20:01:31.201 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T20:01:31.202 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T20:01:31.202 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -qi grub2-tools 2023-10-30T20:01:31.240 INFO:teuthology.orchestra.run.smithi069.stdout:Name : grub2-tools 2023-10-30T20:01:31.240 INFO:teuthology.orchestra.run.smithi069.stdout:Epoch : 1 2023-10-30T20:01:31.241 INFO:teuthology.orchestra.run.smithi069.stdout:Version : 2.06 2023-10-30T20:01:31.241 INFO:teuthology.orchestra.run.smithi069.stdout:Release : 68.el9 2023-10-30T20:01:31.241 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: x86_64 2023-10-30T20:01:31.241 INFO:teuthology.orchestra.run.smithi069.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-30T20:01:31.242 INFO:teuthology.orchestra.run.smithi069.stdout:Group : Unspecified 2023-10-30T20:01:31.242 INFO:teuthology.orchestra.run.smithi069.stdout:Size : 8264254 2023-10-30T20:01:31.242 INFO:teuthology.orchestra.run.smithi069.stdout:License : GPLv3+ 2023-10-30T20:01:31.243 INFO:teuthology.orchestra.run.smithi069.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-30T20:01:31.243 INFO:teuthology.orchestra.run.smithi069.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-30T20:01:31.243 INFO:teuthology.orchestra.run.smithi069.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-30T20:01:31.244 INFO:teuthology.orchestra.run.smithi069.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-30T20:01:31.244 INFO:teuthology.orchestra.run.smithi069.stdout:Packager : builder@centos.org 2023-10-30T20:01:31.244 INFO:teuthology.orchestra.run.smithi069.stdout:Vendor : CentOS 2023-10-30T20:01:31.244 INFO:teuthology.orchestra.run.smithi069.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-30T20:01:31.245 INFO:teuthology.orchestra.run.smithi069.stdout:Summary : Support tools for GRUB. 2023-10-30T20:01:31.245 INFO:teuthology.orchestra.run.smithi069.stdout:Description : 2023-10-30T20:01:31.245 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:01:31.246 INFO:teuthology.orchestra.run.smithi069.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-30T20:01:31.246 INFO:teuthology.orchestra.run.smithi069.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-30T20:01:31.246 INFO:teuthology.orchestra.run.smithi069.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-30T20:01:31.246 INFO:teuthology.orchestra.run.smithi069.stdout:hardware devices. 2023-10-30T20:01:31.247 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:01:31.247 INFO:teuthology.orchestra.run.smithi069.stdout:This subpackage provides tools for support of all platforms. 2023-10-30T20:01:31.248 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-30T20:01:31.249 INFO:teuthology.task.kernel:Updating grub on smithi069 to boot 5.14.0-378.el9.x86_64 2023-10-30T20:01:31.249 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-30T20:01:31.851 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2023-10-30T20:01:33.350 INFO:teuthology.orchestra.run.smithi069.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-30T20:01:33.381 INFO:teuthology.orchestra.run.smithi069.stderr:done 2023-10-30T20:01:33.383 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2023-10-30T20:01:33.399 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.esQYzM4p2Q 2023-10-30T20:01:33.399 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.esQYzM4p2Q 2023-10-30T20:01:33.467 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.esQYzM4p2Q 2023-10-30T20:01:33.604 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.esQYzM4p2Q is 6KB 2023-10-30T20:01:33.651 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.esQYzM4p2Q 2023-10-30T20:01:33.666 DEBUG:teuthology.orchestra.run.smithi069:> sudo /bin/ls /boot/loader/entries || true 2023-10-30T20:01:33.730 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-30T20:01:33.731 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-30T20:01:33.731 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-30T20:01:33.731 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-10-30T20:01:33.732 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-10-30T20:01:33.885 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2023-10-30T20:01:34.978 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T20:01:35.021 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-10-30T20:01:35.428 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T20:01:35.429 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-10-30T20:01:35.429 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T20:01:35.429 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T20:01:35.865 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T20:01:35.865 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:01:35.865 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:01:35.865 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-5.14.0-378.el9.x86_64 2023-10-30T20:01:35.866 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-10-30T20:01:35.866 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-10-30T20:01:35.866 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-10-30T20:01:35.866 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:01:35.866 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:01:36.369 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T20:01:36.549 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T20:01:36.549 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T20:01:36.549 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-10-30T20:01:36.550 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-30T20:01:36.550 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-30T20:01:36.550 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-10-30T20:01:36.550 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2023-10-30T20:01:36.551 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T20:01:36.669 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T20:01:36.670 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T20:01:36.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2023-10-30T20:01:36.704 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2023-10-30T20:01:36.704 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2023-10-30T20:01:36.705 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2023-10-30T20:01:36.705 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 68.el9 2023-10-30T20:01:36.705 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2023-10-30T20:01:36.705 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-30T20:01:36.706 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2023-10-30T20:01:36.706 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8264254 2023-10-30T20:01:36.706 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2023-10-30T20:01:36.706 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-30T20:01:36.706 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-30T20:01:36.707 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-30T20:01:36.707 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-30T20:01:36.707 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2023-10-30T20:01:36.707 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2023-10-30T20:01:36.708 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-30T20:01:36.708 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2023-10-30T20:01:36.708 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2023-10-30T20:01:36.708 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:01:36.708 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-30T20:01:36.709 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-30T20:01:36.709 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-30T20:01:36.709 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2023-10-30T20:01:36.709 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:01:36.709 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2023-10-30T20:01:36.711 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-30T20:01:36.711 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 5.14.0-378.el9.x86_64 2023-10-30T20:01:36.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-30T20:01:37.314 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2023-10-30T20:01:38.675 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-30T20:01:38.707 INFO:teuthology.orchestra.run.smithi183.stderr:done 2023-10-30T20:01:38.710 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2023-10-30T20:01:38.725 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.Zx44hsebs1 2023-10-30T20:01:38.726 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Zx44hsebs1 2023-10-30T20:01:38.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.Zx44hsebs1 2023-10-30T20:01:38.942 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.Zx44hsebs1 is 6KB 2023-10-30T20:01:38.991 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.Zx44hsebs1 2023-10-30T20:01:39.006 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2023-10-30T20:01:39.073 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-30T20:01:39.074 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-30T20:01:39.074 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-30T20:01:39.074 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-10-30T20:01:39.076 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-10-30T20:01:39.234 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2023-10-30T20:02:03.910 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-30T20:02:03.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-10-30T20:02:03.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T20:02:09.238 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-10-30T20:02:09.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2023-10-30T20:02:09.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T20:02:22.458 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2023-10-30T20:02:27.582 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2023-10-30T20:02:31.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-10-30T20:02:31.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T20:02:34.522 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2023-10-30T20:02:36.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2023-10-30T20:02:36.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T20:02:39.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2023-10-30T20:02:46.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-10-30T20:02:46.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T20:02:51.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2023-10-30T20:02:51.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T20:03:04.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2023-10-30T20:03:10.075 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2023-10-30T20:03:19.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-10-30T20:03:19.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T20:03:20.258 DEBUG:teuthology.orchestra.run.smithi069:> true 2023-10-30T20:03:20.627 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-10-30T20:03:20.627 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-378.el9.x86_64"... 2023-10-30T20:03:20.628 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2023-10-30T20:03:20.642 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-378.el9.x86_64 2023-10-30T20:03:20.642 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-10-30T20:03:20.642 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T20:03:20.643 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-30T20:03:21.644 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-30T20:03:21.644 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-30T20:03:21.715 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2023-10-30T20:03:21.753 DEBUG:teuthology.parallel:result is None 2023-10-30T20:03:25.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2023-10-30T20:03:25.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T20:03:25.487 DEBUG:teuthology.orchestra.run.smithi183:> true 2023-10-30T20:03:25.861 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2023-10-30T20:03:25.862 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-378.el9.x86_64"... 2023-10-30T20:03:25.862 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2023-10-30T20:03:25.876 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-378.el9.x86_64 2023-10-30T20:03:25.877 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-10-30T20:03:25.877 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T20:03:25.877 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-30T20:03:26.878 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-10-30T20:03:26.879 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-30T20:03:26.940 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2023-10-30T20:03:26.980 DEBUG:teuthology.parallel:result is None 2023-10-30T20:03:26.980 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-30T20:03:26.988 INFO:teuthology.task.internal:Creating test directory... 2023-10-30T20:03:26.988 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T20:03:26.991 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T20:03:27.007 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-30T20:03:27.014 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-30T20:03:27.066 INFO:teuthology.task.internal:Creating archive directory... 2023-10-30T20:03:27.067 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T20:03:27.069 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T20:03:27.102 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-30T20:03:27.109 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-30T20:03:27.110 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T20:03:27.140 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T20:03:27.170 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T20:03:27.177 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T20:03:27.186 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T20:03:27.190 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T20:03:27.192 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-30T20:03:27.199 INFO:teuthology.task.internal:Configuring sudo... 2023-10-30T20:03:27.200 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T20:03:27.229 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T20:03:27.260 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-30T20:03:27.269 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-30T20:03:27.269 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T20:03:27.299 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T20:03:27.315 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T20:03:27.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T20:03:27.454 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T20:03:27.537 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T20:03:27.581 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:03:27.582 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T20:03:27.651 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T20:03:27.704 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T20:03:27.773 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T20:03:27.828 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T20:03:27.888 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:03:27.888 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T20:03:27.932 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2023-10-30T20:03:27.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2023-10-30T20:03:27.994 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T20:03:28.058 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T20:03:28.538 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-30T20:03:28.546 INFO:teuthology.task.internal:Starting timer... 2023-10-30T20:03:28.547 INFO:teuthology.run_tasks:Running task pcp... 2023-10-30T20:03:28.562 INFO:teuthology.run_tasks:Running task selinux... 2023-10-30T20:03:28.573 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2023-10-30T20:03:28.641 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: 2023-10-30T20:03:28.643 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2023-10-30T20:03:28.724 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2023-10-30T20:03:28.725 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-30T20:03:28.726 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2023-10-30T20:03:28.746 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2023-10-30T20:03:28.746 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2023-10-30T20:03:28.866 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2023-10-30T20:03:28.866 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi069.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2023-10-30T20:03:28.867 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T20:03:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T20:03:28.896 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T20:03:28.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T20:03:28.925 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-30T20:03:28.925 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2023-10-30T20:03:28.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2023-10-30T20:03:29.084 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-30T20:03:29.094 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-30T20:03:29.095 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-30T20:03:29.114 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-30T20:03:29.115 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi069.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-30T20:08:58.576 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi069.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2023-10-30T20:08:58.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-10-30T20:08:58.579 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T20:08:58.661 DEBUG:teuthology.orchestra.run.smithi069:> true 2023-10-30T20:08:58.728 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-10-30T20:08:58.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2023-10-30T20:08:58.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T20:08:58.814 DEBUG:teuthology.orchestra.run.smithi183:> true 2023-10-30T20:08:58.886 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2023-10-30T20:08:58.886 INFO:teuthology.run_tasks:Running task clock... 2023-10-30T20:08:58.898 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-30T20:08:58.899 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T20:08:58.899 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T20:08:58.902 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T20:08:58.902 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T20:08:58.930 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-30T20:08:58.946 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-30T20:08:58.956 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-30T20:08:58.974 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-30T20:08:58.975 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: ntpd: command not found 2023-10-30T20:08:58.988 INFO:teuthology.orchestra.run.smithi069.stdout:506 Cannot talk to daemon 2023-10-30T20:08:59.003 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2023-10-30T20:08:59.004 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-30T20:08:59.017 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2023-10-30T20:08:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-30T20:08:59.034 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-30T20:08:59.050 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-30T20:08:59.063 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2023-10-30T20:08:59.066 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T20:08:59.067 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2023-10-30T20:08:59.067 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T20:08:59.067 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T20:08:59.067 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T20:08:59.067 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T20:08:59.092 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2023-10-30T20:08:59.095 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T20:08:59.096 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2023-10-30T20:08:59.096 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T20:08:59.096 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T20:08:59.096 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T20:08:59.096 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T20:08:59.098 INFO:teuthology.run_tasks:Running task install... 2023-10-30T20:08:59.106 DEBUG:teuthology.task.install:project ceph 2023-10-30T20:08:59.106 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d'}} 2023-10-30T20:08:59.107 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d'} 2023-10-30T20:08:59.107 INFO:teuthology.task.install:Using flavor: default 2023-10-30T20:08:59.112 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-10-30T20:08:59.113 INFO:teuthology.task.install:extra packages: [] 2023-10-30T20:08:59.113 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d', 'tag': None, 'wait_for_package': False} 2023-10-30T20:08:59.114 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T20:08:59.116 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d', 'tag': None, 'wait_for_package': False} 2023-10-30T20:08:59.117 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T20:08:59.276 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/ 2023-10-30T20:08:59.276 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6839.gcc36dd85 2023-10-30T20:08:59.280 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/ 2023-10-30T20:08:59.281 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6839.gcc36dd85 2023-10-30T20:08:59.397 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-30T20:08:59.397 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:08:59.398 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-30T20:08:59.404 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-30T20:08:59.405 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:08:59.405 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-30T20:08:59.426 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-10-30T20:08:59.427 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cc36dd85d142c6fdb77515cad8af108cec98a86d/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-30T20:08:59.433 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-10-30T20:08:59.433 DEBUG:teuthology.orchestra.run.smithi069:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cc36dd85d142c6fdb77515cad8af108cec98a86d/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-30T20:08:59.493 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-30T20:08:59.501 DEBUG:teuthology.orchestra.run.smithi069:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-30T20:08:59.573 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-30T20:08:59.581 DEBUG:teuthology.orchestra.run.smithi069:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-30T20:08:59.639 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2023-10-30T20:08:59.641 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2023-10-30T20:08:59.646 INFO:teuthology.orchestra.run.smithi069.stdout:check_obsoletes = 1 2023-10-30T20:08:59.648 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum clean all 2023-10-30T20:08:59.874 INFO:teuthology.orchestra.run.smithi183.stdout:72 files removed 2023-10-30T20:08:59.895 INFO:teuthology.orchestra.run.smithi069.stdout:72 files removed 2023-10-30T20:08:59.902 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw 2023-10-30T20:08:59.922 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw 2023-10-30T20:09:06.050 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 13 kB/s | 78 kB 00:05 2023-10-30T20:09:06.061 INFO:teuthology.orchestra.run.smithi069.stdout:ceph packages for x86_64 13 kB/s | 78 kB 00:05 2023-10-30T20:09:06.576 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 20 kB/s | 9.4 kB 00:00 2023-10-30T20:09:06.620 INFO:teuthology.orchestra.run.smithi069.stdout:ceph noarch packages 18 kB/s | 9.4 kB 00:00 2023-10-30T20:09:06.968 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2023-10-30T20:09:06.971 INFO:teuthology.orchestra.run.smithi069.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2023-10-30T20:09:07.311 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 21 MB/s | 6.3 MB 00:00 2023-10-30T20:09:07.340 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 19 MB/s | 6.3 MB 00:00 2023-10-30T20:09:08.704 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2023-10-30T20:09:08.790 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2023-10-30T20:09:09.544 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 7.9 MB 00:00 2023-10-30T20:09:11.556 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 7.9 MB 00:02 2023-10-30T20:09:11.699 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 16 MB/s | 18 MB 00:01 2023-10-30T20:09:16.807 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 5.8 MB 00:01 2023-10-30T20:09:18.205 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 15 kB 00:00 2023-10-30T20:09:18.948 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 2.8 MB/s | 18 MB 00:06 2023-10-30T20:09:19.634 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 15 MB/s | 19 MB 00:01 2023-10-30T20:09:24.082 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2023-10-30T20:09:26.394 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 5.8 MB 00:04 2023-10-30T20:09:26.437 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:09:26.440 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:09:26.440 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2023-10-30T20:09:26.440 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:09:26.440 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:09:26.441 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 8.7 M 2023-10-30T20:09:26.441 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2023-10-30T20:09:26.441 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.3 M 2023-10-30T20:09:26.441 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.0 M 2023-10-30T20:09:26.442 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2023-10-30T20:09:26.442 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options 2023-10-30T20:09:26.442 INFO:teuthology.orchestra.run.smithi183.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-30T20:09:26.442 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.2 M 2023-10-30T20:09:26.442 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 19 M 2023-10-30T20:09:26.443 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 24 k 2023-10-30T20:09:26.443 INFO:teuthology.orchestra.run.smithi183.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-30T20:09:26.443 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-30T20:09:26.443 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-30T20:09:26.443 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-30T20:09:26.444 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 715 k 2023-10-30T20:09:26.444 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-30T20:09:26.444 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-30T20:09:26.444 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-30T20:09:26.444 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 472 k 2023-10-30T20:09:26.445 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-30T20:09:26.445 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 4.8 M 2023-10-30T20:09:26.445 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-30T20:09:26.445 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-30T20:09:26.445 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-30T20:09:26.446 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse 2023-10-30T20:09:26.446 INFO:teuthology.orchestra.run.smithi183.stdout: x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 45 k 2023-10-30T20:09:26.446 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 127 k 2023-10-30T20:09:26.446 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 162 k 2023-10-30T20:09:26.446 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-30T20:09:26.446 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 321 k 2023-10-30T20:09:26.447 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 298 k 2023-10-30T20:09:26.447 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 99 k 2023-10-30T20:09:26.447 INFO:teuthology.orchestra.run.smithi183.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-30T20:09:26.447 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-30T20:09:26.447 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:09:26.448 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:09:26.448 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:09:26.448 INFO:teuthology.orchestra.run.smithi183.stdout:Install 28 Packages 2023-10-30T20:09:26.448 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 2 Packages 2023-10-30T20:09:26.448 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:09:26.449 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 55 M 2023-10-30T20:09:26.449 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:09:27.907 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2023-10-30T20:09:28.041 INFO:teuthology.orchestra.run.smithi183.stdout:(1/30): ceph-radosgw-18.0.0-6839.gcc36dd85.el9. 7.9 MB/s | 8.7 MB 00:01 2023-10-30T20:09:28.091 INFO:teuthology.orchestra.run.smithi183.stdout:(2/30): ceph-selinux-18.0.0-6839.gcc36dd85.el9. 483 kB/s | 24 kB 00:00 2023-10-30T20:09:28.191 INFO:teuthology.orchestra.run.smithi183.stdout:(3/30): libcephfs2-18.0.0-6839.gcc36dd85.el9.x8 7.0 MB/s | 715 kB 00:00 2023-10-30T20:09:28.275 INFO:teuthology.orchestra.run.smithi183.stdout:(4/30): libradosstriper1-18.0.0-6839.gcc36dd85. 5.5 MB/s | 472 kB 00:00 2023-10-30T20:09:28.742 INFO:teuthology.orchestra.run.smithi183.stdout:(5/30): ceph-common-18.0.0-6839.gcc36dd85.el9.x 11 MB/s | 19 MB 00:01 2023-10-30T20:09:28.817 INFO:teuthology.orchestra.run.smithi183.stdout:(6/30): librgw2-18.0.0-6839.gcc36dd85.el9.x86_6 8.8 MB/s | 4.8 MB 00:00 2023-10-30T20:09:28.842 INFO:teuthology.orchestra.run.smithi183.stdout:(7/30): python3-ceph-argparse-18.0.0-6839.gcc36 451 kB/s | 45 kB 00:00 2023-10-30T20:09:28.867 INFO:teuthology.orchestra.run.smithi183.stdout:(8/30): python3-ceph-common-18.0.0-6839.gcc36dd 2.5 MB/s | 127 kB 00:00 2023-10-30T20:09:28.893 INFO:teuthology.orchestra.run.smithi183.stdout:(9/30): python3-cephfs-18.0.0-6839.gcc36dd85.el 3.1 MB/s | 162 kB 00:00 2023-10-30T20:09:28.918 INFO:teuthology.orchestra.run.smithi183.stdout:(10/30): python3-rados-18.0.0-6839.gcc36dd85.el 6.2 MB/s | 321 kB 00:00 2023-10-30T20:09:28.943 INFO:teuthology.orchestra.run.smithi183.stdout:(11/30): python3-rbd-18.0.0-6839.gcc36dd85.el9. 5.8 MB/s | 298 kB 00:00 2023-10-30T20:09:28.969 INFO:teuthology.orchestra.run.smithi183.stdout:(12/30): python3-rgw-18.0.0-6839.gcc36dd85.el9. 1.9 MB/s | 99 kB 00:00 2023-10-30T20:09:29.090 INFO:teuthology.orchestra.run.smithi183.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 729 kB/s | 106 kB 00:00 2023-10-30T20:09:29.127 INFO:teuthology.orchestra.run.smithi183.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2023-10-30T20:09:29.169 INFO:teuthology.orchestra.run.smithi183.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 8.0 MB/s | 661 kB 00:00 2023-10-30T20:09:29.194 INFO:teuthology.orchestra.run.smithi183.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2023-10-30T20:09:29.212 INFO:teuthology.orchestra.run.smithi183.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 991 kB/s | 41 kB 00:00 2023-10-30T20:09:29.312 INFO:teuthology.orchestra.run.smithi183.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2023-10-30T20:09:29.378 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 14 MB/s | 19 MB 00:01 2023-10-30T20:09:29.395 INFO:teuthology.orchestra.run.smithi183.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 603 kB/s | 111 kB 00:00 2023-10-30T20:09:29.554 INFO:teuthology.orchestra.run.smithi183.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2023-10-30T20:09:29.604 INFO:teuthology.orchestra.run.smithi183.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 490 kB/s | 24 kB 00:00 2023-10-30T20:09:29.663 INFO:teuthology.orchestra.run.smithi183.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2023-10-30T20:09:29.730 INFO:teuthology.orchestra.run.smithi183.stdout:(23/30): libarrow-9.0.0-9.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2023-10-30T20:09:29.755 INFO:teuthology.orchestra.run.smithi183.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2023-10-30T20:09:29.797 INFO:teuthology.orchestra.run.smithi183.stdout:(25/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 12 MB/s | 837 kB 00:00 2023-10-30T20:09:29.822 INFO:teuthology.orchestra.run.smithi183.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2023-10-30T20:09:29.881 INFO:teuthology.orchestra.run.smithi183.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2023-10-30T20:09:30.173 INFO:teuthology.orchestra.run.smithi183.stdout:(28/30): librados2-18.0.0-6839.gcc36dd85.el9.x8 9.4 MB/s | 3.3 MB 00:00 2023-10-30T20:09:30.256 INFO:teuthology.orchestra.run.smithi183.stdout:(29/30): librbd1-18.0.0-6839.gcc36dd85.el9.x86_ 8.1 MB/s | 3.0 MB 00:00 2023-10-30T20:09:30.440 INFO:teuthology.orchestra.run.smithi183.stdout:(30/30): ceph-base-18.0.0-6839.gcc36dd85.el9.x8 1.5 MB/s | 5.2 MB 00:03 2023-10-30T20:09:30.444 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:09:30.444 INFO:teuthology.orchestra.run.smithi183.stdout:Total 14 MB/s | 55 MB 00:04 2023-10-30T20:09:30.454 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:09:30.486 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:09:30.486 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:09:30.955 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:09:30.955 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:09:32.493 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:09:32.552 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-30T20:09:32.601 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-10-30T20:09:32.654 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-10-30T20:09:32.970 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-10-30T20:09:33.018 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T20:09:33.110 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T20:09:33.194 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/32 2023-10-30T20:09:33.220 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T20:09:33.266 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T20:09:33.286 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-10-30T20:09:33.320 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9. 9/32 2023-10-30T20:09:33.386 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 10/32 2023-10-30T20:09:33.413 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 11/32 2023-10-30T20:09:33.479 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 11/32 2023-10-30T20:09:33.586 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-30T20:09:33.630 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-30T20:09:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-30T20:09:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-30T20:09:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-30T20:09:33.765 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-30T20:09:33.798 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T20:09:34.025 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 9.9 kB/s | 1.7 kB 00:00 2023-10-30T20:09:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-30T20:09:34.142 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 20/32 2023-10-30T20:09:34.241 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 20/32 2023-10-30T20:09:34.272 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 21/32 2023-10-30T20:09:34.309 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-30T20:09:34.739 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-30T20:09:34.762 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 24/32 2023-10-30T20:09:34.816 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 24/32 2023-10-30T20:09:34.917 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 25/32 2023-10-30T20:09:34.947 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x8 26/32 2023-10-30T20:09:36.390 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:09:36.397 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:09:36.397 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-10-30T20:09:36.397 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:09:36.397 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:09:36.397 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 8.7 M 2023-10-30T20:09:36.398 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrading: 2023-10-30T20:09:36.398 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.3 M 2023-10-30T20:09:36.398 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.0 M 2023-10-30T20:09:36.398 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-10-30T20:09:36.398 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options 2023-10-30T20:09:36.399 INFO:teuthology.orchestra.run.smithi069.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-30T20:09:36.399 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.2 M 2023-10-30T20:09:36.399 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 19 M 2023-10-30T20:09:36.399 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 24 k 2023-10-30T20:09:36.400 INFO:teuthology.orchestra.run.smithi069.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-30T20:09:36.400 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-30T20:09:36.400 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-30T20:09:36.400 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-30T20:09:36.400 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 715 k 2023-10-30T20:09:36.401 INFO:teuthology.orchestra.run.smithi069.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-30T20:09:36.401 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-30T20:09:36.401 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-30T20:09:36.401 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 472 k 2023-10-30T20:09:36.401 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-30T20:09:36.402 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 4.8 M 2023-10-30T20:09:36.402 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-30T20:09:36.402 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-30T20:09:36.402 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-30T20:09:36.402 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse 2023-10-30T20:09:36.403 INFO:teuthology.orchestra.run.smithi069.stdout: x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 45 k 2023-10-30T20:09:36.403 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 127 k 2023-10-30T20:09:36.403 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 162 k 2023-10-30T20:09:36.404 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-30T20:09:36.404 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 321 k 2023-10-30T20:09:36.404 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 298 k 2023-10-30T20:09:36.404 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 99 k 2023-10-30T20:09:36.404 INFO:teuthology.orchestra.run.smithi069.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-30T20:09:36.405 INFO:teuthology.orchestra.run.smithi069.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-30T20:09:36.405 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:09:36.405 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:09:36.405 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:09:36.405 INFO:teuthology.orchestra.run.smithi069.stdout:Install 28 Packages 2023-10-30T20:09:36.406 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrade 2 Packages 2023-10-30T20:09:36.406 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:09:36.406 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 55 M 2023-10-30T20:09:36.407 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:09:37.127 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T20:09:37.160 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T20:09:37.268 INFO:teuthology.orchestra.run.smithi069.stdout:(1/30): ceph-base-18.0.0-6839.gcc36dd85.el9.x86 6.7 MB/s | 5.2 MB 00:00 2023-10-30T20:09:37.327 INFO:teuthology.orchestra.run.smithi069.stdout:(2/30): ceph-selinux-18.0.0-6839.gcc36dd85.el9. 414 kB/s | 24 kB 00:00 2023-10-30T20:09:37.452 INFO:teuthology.orchestra.run.smithi069.stdout:(3/30): libcephfs2-18.0.0-6839.gcc36dd85.el9.x8 5.6 MB/s | 715 kB 00:00 2023-10-30T20:09:37.585 INFO:teuthology.orchestra.run.smithi069.stdout:(4/30): ceph-radosgw-18.0.0-6839.gcc36dd85.el9. 7.9 MB/s | 8.7 MB 00:01 2023-10-30T20:09:37.713 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T20:09:37.755 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 28/32 2023-10-30T20:09:37.843 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 28/32 2023-10-30T20:09:37.844 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-30T20:09:37.844 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-30T20:09:37.845 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:09:37.891 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T20:09:42.768 INFO:teuthology.orchestra.run.smithi069.stdout:(5/30): libradosstriper1-18.0.0-6839.gcc36dd85. 89 kB/s | 472 kB 00:05 2023-10-30T20:09:42.886 INFO:teuthology.orchestra.run.smithi069.stdout:(6/30): python3-ceph-argparse-18.0.0-6839.gcc36 387 kB/s | 45 kB 00:00 2023-10-30T20:09:42.986 INFO:teuthology.orchestra.run.smithi069.stdout:(7/30): python3-ceph-common-18.0.0-6839.gcc36dd 1.2 MB/s | 127 kB 00:00 2023-10-30T20:09:43.086 INFO:teuthology.orchestra.run.smithi069.stdout:(8/30): python3-cephfs-18.0.0-6839.gcc36dd85.el 1.6 MB/s | 162 kB 00:00 2023-10-30T20:09:43.361 INFO:teuthology.orchestra.run.smithi069.stdout:(9/30): librgw2-18.0.0-6839.gcc36dd85.el9.x86_6 847 kB/s | 4.8 MB 00:05 2023-10-30T20:09:43.487 INFO:teuthology.orchestra.run.smithi069.stdout:(10/30): python3-rbd-18.0.0-6839.gcc36dd85.el9. 2.3 MB/s | 298 kB 00:00 2023-10-30T20:09:43.553 INFO:teuthology.orchestra.run.smithi069.stdout:(11/30): python3-rgw-18.0.0-6839.gcc36dd85.el9. 1.5 MB/s | 99 kB 00:00 2023-10-30T20:09:43.654 INFO:teuthology.orchestra.run.smithi069.stdout:(12/30): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2023-10-30T20:09:43.688 INFO:teuthology.orchestra.run.smithi069.stdout:(13/30): python3-rados-18.0.0-6839.gcc36dd85.el 535 kB/s | 321 kB 00:00 2023-10-30T20:09:43.713 INFO:teuthology.orchestra.run.smithi069.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.6 MB/s | 159 kB 00:00 2023-10-30T20:09:43.746 INFO:teuthology.orchestra.run.smithi069.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-10-30T20:09:43.772 INFO:teuthology.orchestra.run.smithi069.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-10-30T20:09:43.797 INFO:teuthology.orchestra.run.smithi069.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 827 kB/s | 41 kB 00:00 2023-10-30T20:09:43.897 INFO:teuthology.orchestra.run.smithi069.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 362 kB/s | 45 kB 00:00 2023-10-30T20:09:44.106 INFO:teuthology.orchestra.run.smithi069.stdout:(19/30): ceph-common-18.0.0-6839.gcc36dd85.el9. 2.5 MB/s | 19 MB 00:07 2023-10-30T20:09:44.323 INFO:teuthology.orchestra.run.smithi069.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 211 kB/s | 111 kB 00:00 2023-10-30T20:09:44.423 INFO:teuthology.orchestra.run.smithi069.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 244 kB/s | 24 kB 00:00 2023-10-30T20:09:44.449 INFO:teuthology.orchestra.run.smithi069.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 561 kB/s | 309 kB 00:00 2023-10-30T20:09:44.524 INFO:teuthology.orchestra.run.smithi069.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2023-10-30T20:09:44.549 INFO:teuthology.orchestra.run.smithi069.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 673 kB/s | 67 kB 00:00 2023-10-30T20:09:44.641 INFO:teuthology.orchestra.run.smithi069.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2023-10-30T20:09:44.774 INFO:teuthology.orchestra.run.smithi069.stdout:(26/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.3 MB/s | 837 kB 00:00 2023-10-30T20:09:44.841 INFO:teuthology.orchestra.run.smithi069.stdout:(27/30): libarrow-9.0.0-9.el9.x86_64.rpm 6.2 MB/s | 4.4 MB 00:00 2023-10-30T20:09:44.884 INFO:teuthology.orchestra.run.smithi069.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2023-10-30T20:09:45.691 INFO:teuthology.orchestra.run.smithi069.stdout:(29/30): librbd1-18.0.0-6839.gcc36dd85.el9.x86_ 3.6 MB/s | 3.0 MB 00:00 2023-10-30T20:09:47.383 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T20:09:47.383 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2023-10-30T20:09:47.384 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2023-10-30T20:09:47.384 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2023-10-30T20:09:47.384 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2023-10-30T20:09:47.384 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2023-10-30T20:09:47.385 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2023-10-30T20:09:47.385 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2023-10-30T20:09:47.385 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:09:48.008 INFO:teuthology.orchestra.run.smithi069.stdout:(30/30): librados2-18.0.0-6839.gcc36dd85.el9.x8 1.0 MB/s | 3.3 MB 00:03 2023-10-30T20:09:48.014 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:09:48.015 INFO:teuthology.orchestra.run.smithi069.stdout:Total 4.7 MB/s | 55 MB 00:11 2023-10-30T20:09:48.027 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:09:48.066 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:09:48.066 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:09:48.223 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/32 2023-10-30T20:09:48.247 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/32 2023-10-30T20:09:48.248 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:09:48.248 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-30T20:09:48.248 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T20:09:48.248 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T20:09:48.249 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:09:48.249 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T20:09:48.334 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T20:09:48.334 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T20:09:48.556 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:09:48.556 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:09:49.398 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T20:09:49.398 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/32 2023-10-30T20:09:49.399 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/32 2023-10-30T20:09:49.399 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 3/32 2023-10-30T20:09:49.399 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 4/32 2023-10-30T20:09:49.399 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T20:09:49.399 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 6/32 2023-10-30T20:09:49.400 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T20:09:49.400 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9. 8/32 2023-10-30T20:09:49.400 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x8 9/32 2023-10-30T20:09:49.400 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 10/32 2023-10-30T20:09:49.400 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 11/32 2023-10-30T20:09:49.400 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 12/32 2023-10-30T20:09:49.401 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 13/32 2023-10-30T20:09:49.401 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-30T20:09:49.401 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-30T20:09:49.401 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-30T20:09:49.401 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-30T20:09:49.401 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T20:09:49.402 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-30T20:09:49.402 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-30T20:09:49.402 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-30T20:09:49.402 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-30T20:09:49.402 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-30T20:09:49.403 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-30T20:09:49.403 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-30T20:09:49.403 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-30T20:09:49.403 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-30T20:09:49.403 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-30T20:09:49.404 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T20:09:49.404 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-30T20:09:49.404 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/32 2023-10-30T20:09:49.927 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:09:49.996 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-30T20:09:50.042 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-10-30T20:09:50.096 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-10-30T20:09:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-10-30T20:09:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T20:09:50.565 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T20:09:50.565 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:09:50.566 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2023-10-30T20:09:50.566 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.566 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.566 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:09:50.566 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-30T20:09:50.567 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.567 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.567 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.567 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.567 INFO:teuthology.orchestra.run.smithi183.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-30T20:09:50.568 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-30T20:09:50.568 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-30T20:09:50.568 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-30T20:09:50.568 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.568 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-30T20:09:50.569 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-30T20:09:50.569 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-30T20:09:50.569 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.569 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-30T20:09:50.569 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.570 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-30T20:09:50.570 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-30T20:09:50.570 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-30T20:09:50.570 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.571 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.571 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.571 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-30T20:09:50.571 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.571 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.571 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:09:50.572 INFO:teuthology.orchestra.run.smithi183.stdout: re2-1:20211101-3.el9.x86_64 2023-10-30T20:09:50.572 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-30T20:09:50.572 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:09:50.572 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:09:50.577 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T20:09:50.663 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/32 2023-10-30T20:09:50.688 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T20:09:50.758 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T20:09:50.788 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-10-30T20:09:50.832 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9. 9/32 2023-10-30T20:09:50.897 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-test 2023-10-30T20:09:50.898 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 10/32 2023-10-30T20:09:50.924 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 11/32 2023-10-30T20:09:50.998 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 11/32 2023-10-30T20:09:51.104 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-30T20:09:51.142 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-30T20:09:51.182 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-30T20:09:51.222 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-30T20:09:51.249 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-30T20:09:51.285 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-30T20:09:51.309 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T20:09:51.437 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:27 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:09:51.578 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:09:51.579 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:09:51.579 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2023-10-30T20:09:51.579 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:09:51.580 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:09:51.580 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 41 M 2023-10-30T20:09:51.580 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2023-10-30T20:09:51.580 INFO:teuthology.orchestra.run.smithi183.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-30T20:09:51.581 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 163 k 2023-10-30T20:09:51.581 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-30T20:09:51.581 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-30T20:09:51.581 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-30T20:09:51.581 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:09:51.582 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:09:51.582 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:09:51.582 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2023-10-30T20:09:51.582 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:09:51.583 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 42 M 2023-10-30T20:09:51.583 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 172 M 2023-10-30T20:09:51.583 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:09:51.627 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-30T20:09:51.664 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 20/32 2023-10-30T20:09:51.753 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 20/32 2023-10-30T20:09:51.785 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 21/32 2023-10-30T20:09:51.821 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-30T20:09:52.005 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2023-10-30T20:09:52.056 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2023-10-30T20:09:52.089 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-10-30T20:09:52.231 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): jq-1.6-15.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2023-10-30T20:09:52.256 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): libcephsqlite-18.0.0-6839.gcc36dd85.el9. 402 kB/s | 163 kB 00:00 2023-10-30T20:09:52.257 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-30T20:09:52.292 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 24/32 2023-10-30T20:09:52.362 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 24/32 2023-10-30T20:09:52.430 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 25/32 2023-10-30T20:09:52.461 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x8 26/32 2023-10-30T20:09:54.551 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T20:09:54.590 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T20:09:55.140 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T20:09:55.218 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 28/32 2023-10-30T20:09:55.310 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 28/32 2023-10-30T20:09:55.310 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-30T20:09:55.310 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-30T20:09:55.311 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:09:55.336 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T20:09:55.831 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): ceph-test-18.0.0-6839.gcc36dd85.el9.x86_ 10 MB/s | 41 MB 00:03 2023-10-30T20:09:55.834 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:09:55.834 INFO:teuthology.orchestra.run.smithi183.stdout:Total 9.8 MB/s | 42 MB 00:04 2023-10-30T20:09:55.848 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:09:55.870 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:09:55.870 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:09:56.232 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:09:56.233 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:09:56.798 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:09:56.882 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-10-30T20:09:56.932 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-10-30T20:09:56.964 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T20:09:57.014 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-10-30T20:09:57.042 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/6 2023-10-30T20:10:00.903 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/6 2023-10-30T20:10:00.974 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/6 2023-10-30T20:10:03.209 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/6 2023-10-30T20:10:03.209 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/6 2023-10-30T20:10:03.209 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/6 2023-10-30T20:10:03.209 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T20:10:03.210 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-10-30T20:10:03.210 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-10-30T20:10:03.628 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-10-30T20:10:03.628 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:03.628 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:10:03.628 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:03.628 INFO:teuthology.orchestra.run.smithi183.stdout: jq-1.6-15.el9.x86_64 2023-10-30T20:10:03.629 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:03.629 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-30T20:10:03.629 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-30T20:10:03.629 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-30T20:10:03.629 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:03.630 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:03.932 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph 2023-10-30T20:10:04.484 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:40 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:04.532 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T20:10:04.533 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /sys 2023-10-30T20:10:04.533 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /proc 2023-10-30T20:10:04.533 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /mnt 2023-10-30T20:10:04.533 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /var/tmp 2023-10-30T20:10:04.534 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /home 2023-10-30T20:10:04.534 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /root 2023-10-30T20:10:04.534 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /tmp 2023-10-30T20:10:04.534 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:04.628 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:04.630 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2023-10-30T20:10:04.631 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2023-10-30T20:10:04.631 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2023-10-30T20:10:04.631 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:10:04.631 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 6.5 k 2023-10-30T20:10:04.632 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2023-10-30T20:10:04.632 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 2.1 M 2023-10-30T20:10:04.632 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.5 M 2023-10-30T20:10:04.632 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 243 k 2023-10-30T20:10:04.633 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 4.4 M 2023-10-30T20:10:04.633 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 18 M 2023-10-30T20:10:04.633 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-30T20:10:04.633 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-30T20:10:04.633 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-30T20:10:04.634 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-30T20:10:04.634 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-30T20:10:04.634 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-30T20:10:04.634 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-30T20:10:04.635 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-30T20:10:04.635 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-30T20:10:04.635 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-30T20:10:04.635 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-30T20:10:04.635 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-30T20:10:04.636 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-30T20:10:04.636 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-30T20:10:04.636 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-30T20:10:04.636 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-30T20:10:04.636 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-30T20:10:04.637 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-30T20:10:04.637 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-30T20:10:04.637 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-30T20:10:04.637 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-30T20:10:04.638 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-30T20:10:04.638 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-30T20:10:04.638 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-10-30T20:10:04.638 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-30T20:10:04.638 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-30T20:10:04.639 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-30T20:10:04.639 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:04.639 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:10:04.639 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2023-10-30T20:10:04.640 INFO:teuthology.orchestra.run.smithi183.stdout:Install 33 Packages 2023-10-30T20:10:04.640 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:04.641 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 30 M 2023-10-30T20:10:04.641 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 110 M 2023-10-30T20:10:04.641 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:10:04.999 INFO:teuthology.orchestra.run.smithi183.stdout:(1/33): ceph-18.0.0-6839.gcc36dd85.el9.x86_64.r 38 kB/s | 6.5 kB 00:00 2023-10-30T20:10:05.241 INFO:teuthology.orchestra.run.smithi183.stdout:(2/33): ceph-mgr-18.0.0-6839.gcc36dd85.el9.x86_ 3.5 MB/s | 1.5 MB 00:00 2023-10-30T20:10:05.283 INFO:teuthology.orchestra.run.smithi183.stdout:(3/33): ceph-mds-18.0.0-6839.gcc36dd85.el9.x86_ 4.6 MB/s | 2.1 MB 00:00 2023-10-30T20:10:05.317 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/32 2023-10-30T20:10:05.340 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/32 2023-10-30T20:10:05.340 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:10:05.341 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-30T20:10:05.341 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T20:10:05.341 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T20:10:05.341 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:05.342 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T20:10:05.383 INFO:teuthology.orchestra.run.smithi183.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6839.gcc36 2.4 MB/s | 243 kB 00:00 2023-10-30T20:10:05.419 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T20:10:05.419 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T20:10:05.508 INFO:teuthology.orchestra.run.smithi183.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2023-10-30T20:10:05.584 INFO:teuthology.orchestra.run.smithi183.stdout:(6/33): ceph-mon-18.0.0-6839.gcc36dd85.el9.x86_ 7.5 MB/s | 4.4 MB 00:00 2023-10-30T20:10:05.601 INFO:teuthology.orchestra.run.smithi183.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 342 kB/s | 32 kB 00:00 2023-10-30T20:10:05.618 INFO:teuthology.orchestra.run.smithi183.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-10-30T20:10:05.643 INFO:teuthology.orchestra.run.smithi183.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-10-30T20:10:05.660 INFO:teuthology.orchestra.run.smithi183.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2023-10-30T20:10:05.685 INFO:teuthology.orchestra.run.smithi183.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2023-10-30T20:10:05.711 INFO:teuthology.orchestra.run.smithi183.stdout:(12/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-10-30T20:10:05.736 INFO:teuthology.orchestra.run.smithi183.stdout:(13/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-10-30T20:10:05.844 INFO:teuthology.orchestra.run.smithi183.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 471 kB/s | 51 kB 00:00 2023-10-30T20:10:05.928 INFO:teuthology.orchestra.run.smithi183.stdout:(15/33): python3-cryptography-36.0.1-4.el9.x86_ 5.7 MB/s | 1.2 MB 00:00 2023-10-30T20:10:06.278 INFO:teuthology.orchestra.run.smithi183.stdout:(16/33): ceph-osd-18.0.0-6839.gcc36dd85.el9.x86 17 MB/s | 18 MB 00:01 2023-10-30T20:10:06.537 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T20:10:06.537 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/32 2023-10-30T20:10:06.538 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/32 2023-10-30T20:10:06.538 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 3/32 2023-10-30T20:10:06.538 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 4/32 2023-10-30T20:10:06.538 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T20:10:06.538 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 6/32 2023-10-30T20:10:06.539 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T20:10:06.539 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9. 8/32 2023-10-30T20:10:06.540 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x8 9/32 2023-10-30T20:10:06.540 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 10/32 2023-10-30T20:10:06.541 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 11/32 2023-10-30T20:10:06.541 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 12/32 2023-10-30T20:10:06.541 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 13/32 2023-10-30T20:10:06.541 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-30T20:10:06.541 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-30T20:10:06.542 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-30T20:10:06.542 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-30T20:10:06.542 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T20:10:06.542 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-30T20:10:06.543 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-30T20:10:06.543 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-30T20:10:06.543 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-30T20:10:06.543 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-30T20:10:06.543 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-30T20:10:06.543 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-30T20:10:06.544 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-30T20:10:06.544 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-30T20:10:06.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-30T20:10:06.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T20:10:06.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-30T20:10:06.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/32 2023-10-30T20:10:06.553 INFO:teuthology.orchestra.run.smithi183.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 61 kB/s | 43 kB 00:00 2023-10-30T20:10:06.670 INFO:teuthology.orchestra.run.smithi183.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2023-10-30T20:10:06.779 INFO:teuthology.orchestra.run.smithi183.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2023-10-30T20:10:06.813 INFO:teuthology.orchestra.run.smithi183.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 199 kB/s | 172 kB 00:00 2023-10-30T20:10:06.880 INFO:teuthology.orchestra.run.smithi183.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2023-10-30T20:10:06.938 INFO:teuthology.orchestra.run.smithi183.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2023-10-30T20:10:06.997 INFO:teuthology.orchestra.run.smithi183.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 168 kB/s | 20 kB 00:00 2023-10-30T20:10:07.047 INFO:teuthology.orchestra.run.smithi183.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 427 kB/s | 46 kB 00:00 2023-10-30T20:10:07.222 INFO:teuthology.orchestra.run.smithi183.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 350 kB/s | 79 kB 00:00 2023-10-30T20:10:07.339 INFO:teuthology.orchestra.run.smithi183.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 338 kB/s | 358 kB 00:01 2023-10-30T20:10:07.356 INFO:teuthology.orchestra.run.smithi183.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 123 kB/s | 16 kB 00:00 2023-10-30T20:10:07.465 INFO:teuthology.orchestra.run.smithi183.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 330 kB/s | 36 kB 00:00 2023-10-30T20:10:07.540 INFO:teuthology.orchestra.run.smithi183.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 448 kB/s | 90 kB 00:00 2023-10-30T20:10:07.574 INFO:teuthology.orchestra.run.smithi183.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 517 kB/s | 272 kB 00:00 2023-10-30T20:10:07.682 INFO:teuthology.orchestra.run.smithi183.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2023-10-30T20:10:07.718 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T20:10:07.718 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:07.718 INFO:teuthology.orchestra.run.smithi069.stdout:Upgraded: 2023-10-30T20:10:07.718 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.719 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.719 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:10:07.719 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-30T20:10:07.719 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.720 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.720 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.720 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.720 INFO:teuthology.orchestra.run.smithi069.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-30T20:10:07.720 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-30T20:10:07.721 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-30T20:10:07.721 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-30T20:10:07.721 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.721 INFO:teuthology.orchestra.run.smithi069.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-30T20:10:07.721 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-30T20:10:07.722 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-30T20:10:07.722 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.722 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-30T20:10:07.722 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.722 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-30T20:10:07.723 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-30T20:10:07.723 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-30T20:10:07.723 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.723 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.723 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.724 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-30T20:10:07.724 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.724 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.724 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:07.724 INFO:teuthology.orchestra.run.smithi069.stdout: re2-1:20211101-3.el9.x86_64 2023-10-30T20:10:07.725 INFO:teuthology.orchestra.run.smithi069.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-30T20:10:07.725 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:07.725 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:10:07.816 INFO:teuthology.orchestra.run.smithi183.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 655 kB/s | 230 kB 00:00 2023-10-30T20:10:08.050 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-test 2023-10-30T20:10:08.124 INFO:teuthology.orchestra.run.smithi183.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 732 kB/s | 427 kB 00:00 2023-10-30T20:10:08.129 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:08.129 INFO:teuthology.orchestra.run.smithi183.stdout:Total 8.6 MB/s | 30 MB 00:03 2023-10-30T20:10:08.162 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:10:08.202 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:10:08.202 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:10:08.570 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:10:08.571 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:10:08.587 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:35 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:10:08.734 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:10:08.735 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:10:08.735 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-10-30T20:10:08.736 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:10:08.736 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:10:08.736 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 41 M 2023-10-30T20:10:08.736 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-10-30T20:10:08.736 INFO:teuthology.orchestra.run.smithi069.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-30T20:10:08.737 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 163 k 2023-10-30T20:10:08.737 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-30T20:10:08.737 INFO:teuthology.orchestra.run.smithi069.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-30T20:10:08.737 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-30T20:10:08.737 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:08.738 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:10:08.738 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:10:08.738 INFO:teuthology.orchestra.run.smithi069.stdout:Install 6 Packages 2023-10-30T20:10:08.738 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:08.739 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 42 M 2023-10-30T20:10:08.739 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 172 M 2023-10-30T20:10:08.739 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:10:08.943 INFO:teuthology.orchestra.run.smithi069.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2023-10-30T20:10:08.985 INFO:teuthology.orchestra.run.smithi069.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2023-10-30T20:10:09.019 INFO:teuthology.orchestra.run.smithi069.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-10-30T20:10:09.119 INFO:teuthology.orchestra.run.smithi069.stdout:(4/6): libcephsqlite-18.0.0-6839.gcc36dd85.el9. 514 kB/s | 163 kB 00:00 2023-10-30T20:10:09.177 INFO:teuthology.orchestra.run.smithi069.stdout:(5/6): jq-1.6-15.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-10-30T20:10:09.807 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:10:09.877 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-30T20:10:09.942 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-30T20:10:10.034 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-30T20:10:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-30T20:10:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-30T20:10:10.244 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-30T20:10:10.282 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-30T20:10:10.349 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-30T20:10:10.403 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-10-30T20:10:10.442 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-30T20:10:10.491 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T20:10:10.555 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T20:10:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-30T20:10:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-30T20:10:10.844 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-30T20:10:10.861 INFO:teuthology.orchestra.run.smithi069.stdout:(6/6): ceph-test-18.0.0-6839.gcc36dd85.el9.x86_ 20 MB/s | 41 MB 00:02 2023-10-30T20:10:10.864 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:10.864 INFO:teuthology.orchestra.run.smithi069.stdout:Total 20 MB/s | 42 MB 00:02 2023-10-30T20:10:10.878 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-30T20:10:10.879 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:10:10.900 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:10:10.900 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:10:10.908 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-30T20:10:10.984 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-30T20:10:11.020 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-30T20:10:11.054 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-30T20:10:11.095 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-30T20:10:11.130 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-30T20:10:11.217 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-30T20:10:11.252 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:10:11.253 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:10:11.260 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-30T20:10:11.332 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-30T20:10:11.468 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-30T20:10:11.585 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T20:10:11.740 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9. 28/33 2023-10-30T20:10:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/33 2023-10-30T20:10:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/33 2023-10-30T20:10:11.791 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:10:11.791 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-30T20:10:11.791 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T20:10:11.791 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T20:10:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:10:11.887 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-10-30T20:10:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-10-30T20:10:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T20:10:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-10-30T20:10:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/6 2023-10-30T20:10:13.359 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/33 2023-10-30T20:10:13.386 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/33 2023-10-30T20:10:13.386 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:10:13.386 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-30T20:10:13.387 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T20:10:13.387 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T20:10:13.387 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/33 2023-10-30T20:10:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/33 2023-10-30T20:10:13.858 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:10:13.858 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-30T20:10:13.858 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T20:10:13.859 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T20:10:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:14.060 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 32/33 2023-10-30T20:10:14.082 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 32/33 2023-10-30T20:10:14.083 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:10:14.083 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-30T20:10:14.083 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T20:10:14.084 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T20:10:14.084 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:14.149 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 33/33 2023-10-30T20:10:15.811 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 33/33 2023-10-30T20:10:15.811 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/33 2023-10-30T20:10:15.812 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/33 2023-10-30T20:10:15.812 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 3/33 2023-10-30T20:10:15.812 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 4/33 2023-10-30T20:10:15.812 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/33 2023-10-30T20:10:15.812 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9. 6/33 2023-10-30T20:10:15.813 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-30T20:10:15.813 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-30T20:10:15.813 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-30T20:10:15.813 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-30T20:10:15.813 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T20:10:15.814 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T20:10:15.814 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-30T20:10:15.814 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-10-30T20:10:15.814 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-30T20:10:15.814 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-30T20:10:15.815 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-30T20:10:15.815 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-30T20:10:15.815 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-30T20:10:15.815 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-30T20:10:15.815 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-30T20:10:15.816 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-30T20:10:15.816 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-30T20:10:15.817 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-30T20:10:15.817 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-30T20:10:15.817 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-30T20:10:15.817 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T20:10:15.817 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-30T20:10:15.818 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-30T20:10:15.819 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-30T20:10:15.819 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-30T20:10:15.819 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-30T20:10:15.870 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/6 2023-10-30T20:10:15.954 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/6 2023-10-30T20:10:16.679 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-30T20:10:16.679 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:16.680 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:10:16.680 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:16.680 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:16.680 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:16.680 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:16.681 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:16.681 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:16.681 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-30T20:10:16.681 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-30T20:10:16.681 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-30T20:10:16.682 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-30T20:10:16.682 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-30T20:10:16.682 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-30T20:10:16.682 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-30T20:10:16.682 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-30T20:10:16.683 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-30T20:10:16.683 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-30T20:10:16.683 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-30T20:10:16.683 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-30T20:10:16.683 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-30T20:10:16.684 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-30T20:10:16.684 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-30T20:10:16.684 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2023-10-30T20:10:16.684 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-30T20:10:16.684 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-30T20:10:16.685 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-30T20:10:16.685 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-30T20:10:16.685 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-30T20:10:16.686 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-30T20:10:16.686 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-30T20:10:16.686 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-10-30T20:10:16.686 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-30T20:10:16.686 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-30T20:10:16.687 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-30T20:10:16.687 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:16.687 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:16.964 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-base 2023-10-30T20:10:17.511 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:53 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:17.604 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:10:17.653 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:17.654 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2023-10-30T20:10:17.654 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:17.710 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install cephadm 2023-10-30T20:10:18.025 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/6 2023-10-30T20:10:18.026 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/6 2023-10-30T20:10:18.026 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/6 2023-10-30T20:10:18.026 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T20:10:18.026 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-10-30T20:10:18.026 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-10-30T20:10:18.254 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:54 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:18.390 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-10-30T20:10:18.391 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:18.391 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:10:18.391 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:18.391 INFO:teuthology.orchestra.run.smithi069.stdout: jq-1.6-15.el9.x86_64 2023-10-30T20:10:18.392 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:18.392 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-30T20:10:18.392 INFO:teuthology.orchestra.run.smithi069.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-30T20:10:18.392 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-30T20:10:18.392 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:18.393 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:10:18.394 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:18.394 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:18.394 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2023-10-30T20:10:18.394 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:18.394 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:10:18.395 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 270 k 2023-10-30T20:10:18.395 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:18.395 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:10:18.395 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:18.395 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:10:18.396 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:18.396 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 270 k 2023-10-30T20:10:18.396 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 265 k 2023-10-30T20:10:18.396 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:10:18.675 INFO:teuthology.orchestra.run.smithi183.stdout:cephadm-18.0.0-6839.gcc36dd85.el9.noarch.rpm 959 kB/s | 270 kB 00:00 2023-10-30T20:10:18.676 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:18.676 INFO:teuthology.orchestra.run.smithi183.stdout:Total 954 kB/s | 270 kB 00:00 2023-10-30T20:10:18.676 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:10:18.681 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:10:18.681 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:10:18.684 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:10:18.685 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:10:18.709 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph 2023-10-30T20:10:18.753 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:10:19.105 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T20:10:19.188 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T20:10:19.251 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:46 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:10:19.403 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:10:19.405 INFO:teuthology.orchestra.run.smithi069.stdout:========================================================================================= 2023-10-30T20:10:19.406 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-10-30T20:10:19.406 INFO:teuthology.orchestra.run.smithi069.stdout:========================================================================================= 2023-10-30T20:10:19.407 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:10:19.407 INFO:teuthology.orchestra.run.smithi069.stdout: ceph x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 6.5 k 2023-10-30T20:10:19.407 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-10-30T20:10:19.408 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 2.1 M 2023-10-30T20:10:19.408 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.5 M 2023-10-30T20:10:19.408 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 243 k 2023-10-30T20:10:19.409 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 4.4 M 2023-10-30T20:10:19.409 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 18 M 2023-10-30T20:10:19.409 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-30T20:10:19.409 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-30T20:10:19.409 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-30T20:10:19.410 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-30T20:10:19.410 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-30T20:10:19.410 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-30T20:10:19.410 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-30T20:10:19.411 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-30T20:10:19.411 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-30T20:10:19.411 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-30T20:10:19.411 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-30T20:10:19.411 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-30T20:10:19.412 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-30T20:10:19.412 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-30T20:10:19.412 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-30T20:10:19.412 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-30T20:10:19.412 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-30T20:10:19.412 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-30T20:10:19.413 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-30T20:10:19.413 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-30T20:10:19.413 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-30T20:10:19.413 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-30T20:10:19.413 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-30T20:10:19.414 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-10-30T20:10:19.414 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-30T20:10:19.414 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-30T20:10:19.414 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-30T20:10:19.414 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:19.415 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:10:19.415 INFO:teuthology.orchestra.run.smithi069.stdout:========================================================================================= 2023-10-30T20:10:19.415 INFO:teuthology.orchestra.run.smithi069.stdout:Install 33 Packages 2023-10-30T20:10:19.415 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:19.416 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 30 M 2023-10-30T20:10:19.416 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 110 M 2023-10-30T20:10:19.416 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:10:19.543 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T20:10:19.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T20:10:19.852 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:19.852 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:10:19.852 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:19.852 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:19.853 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:20.046 INFO:teuthology.orchestra.run.smithi069.stdout:(1/33): ceph-18.0.0-6839.gcc36dd85.el9.x86_64.r 40 kB/s | 6.5 kB 00:00 2023-10-30T20:10:20.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-immutable-object-cache 2023-10-30T20:10:20.329 INFO:teuthology.orchestra.run.smithi069.stdout:(2/33): ceph-mds-18.0.0-6839.gcc36dd85.el9.x86_ 4.7 MB/s | 2.1 MB 00:00 2023-10-30T20:10:20.529 INFO:teuthology.orchestra.run.smithi069.stdout:(3/33): ceph-mon-18.0.0-6839.gcc36dd85.el9.x86_ 9.1 MB/s | 4.4 MB 00:00 2023-10-30T20:10:20.613 INFO:teuthology.orchestra.run.smithi069.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6839.gcc36 2.8 MB/s | 243 kB 00:00 2023-10-30T20:10:20.629 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:56 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:20.647 INFO:teuthology.orchestra.run.smithi069.stdout:(5/33): ceph-mgr-18.0.0-6839.gcc36dd85.el9.x86_ 1.9 MB/s | 1.5 MB 00:00 2023-10-30T20:10:20.755 INFO:teuthology.orchestra.run.smithi069.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 292 kB/s | 32 kB 00:00 2023-10-30T20:10:20.774 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:20.775 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:10:20.775 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:20.776 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:10:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 141 k 2023-10-30T20:10:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:20.776 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:10:20.776 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:20.777 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:10:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:20.777 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 141 k 2023-10-30T20:10:20.777 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 395 k 2023-10-30T20:10:20.777 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:10:20.780 INFO:teuthology.orchestra.run.smithi069.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 863 kB/s | 144 kB 00:00 2023-10-30T20:10:20.806 INFO:teuthology.orchestra.run.smithi069.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2023-10-30T20:10:20.831 INFO:teuthology.orchestra.run.smithi069.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-10-30T20:10:20.856 INFO:teuthology.orchestra.run.smithi069.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2023-10-30T20:10:20.882 INFO:teuthology.orchestra.run.smithi069.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-10-30T20:10:20.907 INFO:teuthology.orchestra.run.smithi069.stdout:(12/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-10-30T20:10:20.932 INFO:teuthology.orchestra.run.smithi069.stdout:(13/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2023-10-30T20:10:21.033 INFO:teuthology.orchestra.run.smithi069.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 508 kB/s | 51 kB 00:00 2023-10-30T20:10:21.098 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-immutable-object-cache-18.0.0-6839.gcc36dd 438 kB/s | 141 kB 00:00 2023-10-30T20:10:21.099 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:21.099 INFO:teuthology.orchestra.run.smithi183.stdout:Total 436 kB/s | 141 kB 00:00 2023-10-30T20:10:21.099 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:10:21.106 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:10:21.107 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:10:21.141 INFO:teuthology.orchestra.run.smithi069.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 401 kB/s | 43 kB 00:00 2023-10-30T20:10:21.157 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:10:21.158 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:10:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:10:21.342 INFO:teuthology.orchestra.run.smithi069.stdout:(16/33): ceph-osd-18.0.0-6839.gcc36dd85.el9.x86 17 MB/s | 18 MB 00:01 2023-10-30T20:10:21.357 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T20:10:21.367 INFO:teuthology.orchestra.run.smithi069.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 762 kB/s | 172 kB 00:00 2023-10-30T20:10:21.381 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T20:10:21.382 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:10:21.382 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-30T20:10:21.382 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:21.400 INFO:teuthology.orchestra.run.smithi069.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 323 kB/s | 11 kB 00:00 2023-10-30T20:10:21.434 INFO:teuthology.orchestra.run.smithi069.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 528 kB/s | 18 kB 00:00 2023-10-30T20:10:21.476 INFO:teuthology.orchestra.run.smithi069.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2023-10-30T20:10:21.501 INFO:teuthology.orchestra.run.smithi069.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2023-10-30T20:10:21.527 INFO:teuthology.orchestra.run.smithi069.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2023-10-30T20:10:21.552 INFO:teuthology.orchestra.run.smithi069.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2023-10-30T20:10:21.577 INFO:teuthology.orchestra.run.smithi069.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2023-10-30T20:10:21.602 INFO:teuthology.orchestra.run.smithi069.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2023-10-30T20:10:21.653 INFO:teuthology.orchestra.run.smithi069.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2023-10-30T20:10:21.678 INFO:teuthology.orchestra.run.smithi069.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 219 kB/s | 16 kB 00:00 2023-10-30T20:10:21.704 INFO:teuthology.orchestra.run.smithi069.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2023-10-30T20:10:21.728 INFO:teuthology.orchestra.run.smithi069.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2023-10-30T20:10:21.754 INFO:teuthology.orchestra.run.smithi069.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2023-10-30T20:10:21.804 INFO:teuthology.orchestra.run.smithi069.stdout:(31/33): python3-cryptography-36.0.1-4.el9.x86_ 1.4 MB/s | 1.2 MB 00:00 2023-10-30T20:10:21.829 INFO:teuthology.orchestra.run.smithi069.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2023-10-30T20:10:21.854 INFO:teuthology.orchestra.run.smithi069.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2023-10-30T20:10:21.858 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:21.859 INFO:teuthology.orchestra.run.smithi069.stdout:Total 12 MB/s | 30 MB 00:02 2023-10-30T20:10:21.891 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:10:21.930 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:10:21.931 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:10:22.290 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:10:22.291 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:10:22.410 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T20:10:22.411 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:22.411 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:10:22.411 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:22.412 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:22.412 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:22.643 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr 2023-10-30T20:10:23.224 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:59 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:23.315 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:10:23.363 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:23.365 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2023-10-30T20:10:23.365 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:23.425 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-dashboard 2023-10-30T20:10:23.538 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:10:23.599 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-30T20:10:23.631 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-30T20:10:23.664 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-30T20:10:23.755 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-30T20:10:23.834 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-30T20:10:23.875 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-30T20:10:23.913 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-30T20:10:23.970 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:59 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:23.981 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-30T20:10:24.035 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-10-30T20:10:24.073 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-30T20:10:24.112 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:24.113 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:24.114 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2023-10-30T20:10:24.114 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:24.114 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:10:24.114 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 1.8 M 2023-10-30T20:10:24.114 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2023-10-30T20:10:24.115 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 23 k 2023-10-30T20:10:24.115 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 15 k 2023-10-30T20:10:24.115 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-30T20:10:24.115 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-30T20:10:24.115 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-30T20:10:24.116 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2023-10-30T20:10:24.116 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-30T20:10:24.116 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:24.116 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:10:24.116 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:24.116 INFO:teuthology.orchestra.run.smithi183.stdout:Install 7 Packages 2023-10-30T20:10:24.117 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:24.117 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.1 M 2023-10-30T20:10:24.118 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 8.6 M 2023-10-30T20:10:24.118 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:10:24.122 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T20:10:24.191 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T20:10:24.389 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-30T20:10:24.436 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-30T20:10:24.474 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-30T20:10:24.509 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-30T20:10:24.519 INFO:teuthology.orchestra.run.smithi183.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6839.gcc3 107 kB/s | 23 kB 00:00 2023-10-30T20:10:24.539 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-30T20:10:24.569 INFO:teuthology.orchestra.run.smithi183.stdout:(2/7): ceph-prometheus-alerts-18.0.0-6839.gcc36 57 kB/s | 15 kB 00:00 2023-10-30T20:10:24.616 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-30T20:10:24.650 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-30T20:10:24.683 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-30T20:10:24.694 INFO:teuthology.orchestra.run.smithi183.stdout:(3/7): ceph-mgr-dashboard-18.0.0-6839.gcc36dd85 4.5 MB/s | 1.8 MB 00:00 2023-10-30T20:10:24.716 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-30T20:10:24.719 INFO:teuthology.orchestra.run.smithi183.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 45 kB/s | 9.0 kB 00:00 2023-10-30T20:10:24.751 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-30T20:10:24.839 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-30T20:10:24.870 INFO:teuthology.orchestra.run.smithi183.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 176 kB/s | 31 kB 00:00 2023-10-30T20:10:24.873 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-30T20:10:24.895 INFO:teuthology.orchestra.run.smithi183.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 126 kB/s | 41 kB 00:00 2023-10-30T20:10:24.945 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-30T20:10:25.083 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-30T20:10:25.187 INFO:teuthology.orchestra.run.smithi183.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 403 kB/s | 188 kB 00:00 2023-10-30T20:10:25.189 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:25.190 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.9 MB/s | 2.1 MB 00:01 2023-10-30T20:10:25.190 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:10:25.199 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:10:25.199 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:10:25.210 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T20:10:25.270 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:10:25.271 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:10:25.377 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9. 28/33 2023-10-30T20:10:25.403 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/33 2023-10-30T20:10:25.428 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/33 2023-10-30T20:10:25.428 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:10:25.428 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-30T20:10:25.429 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T20:10:25.429 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T20:10:25.429 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:25.572 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:10:25.630 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-30T20:10:25.663 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-30T20:10:25.726 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-30T20:10:25.758 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-30T20:10:25.787 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.n 5/7 2023-10-30T20:10:26.026 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9. 6/7 2023-10-30T20:10:26.057 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 7/7 2023-10-30T20:10:26.622 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 7/7 2023-10-30T20:10:26.622 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9. 1/7 2023-10-30T20:10:26.622 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 2/7 2023-10-30T20:10:26.622 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.n 3/7 2023-10-30T20:10:26.623 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-30T20:10:26.623 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-30T20:10:26.623 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-30T20:10:26.951 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-30T20:10:26.952 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:26.952 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:10:26.952 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:26.952 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:26.953 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:26.953 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-30T20:10:26.953 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-30T20:10:26.953 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-30T20:10:26.953 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-30T20:10:26.954 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:26.954 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/33 2023-10-30T20:10:27.032 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/33 2023-10-30T20:10:27.032 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:10:27.032 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-30T20:10:27.032 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T20:10:27.033 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T20:10:27.033 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:27.183 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-30T20:10:27.472 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/33 2023-10-30T20:10:27.495 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/33 2023-10-30T20:10:27.495 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:10:27.495 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-30T20:10:27.495 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T20:10:27.495 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T20:10:27.496 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:27.706 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 32/33 2023-10-30T20:10:27.729 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 32/33 2023-10-30T20:10:27.729 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:10:27.729 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-30T20:10:27.729 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T20:10:27.730 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T20:10:27.730 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:27.733 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:03 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:27.796 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 33/33 2023-10-30T20:10:27.876 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:27.877 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2023-10-30T20:10:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2023-10-30T20:10:27.877 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2023-10-30T20:10:27.878 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:10:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 7.4 M 2023-10-30T20:10:27.878 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2023-10-30T20:10:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-30T20:10:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-30T20:10:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-30T20:10:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-30T20:10:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-30T20:10:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-30T20:10:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-30T20:10:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-30T20:10:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-30T20:10:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-30T20:10:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-30T20:10:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:27.881 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:10:27.881 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2023-10-30T20:10:27.881 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2023-10-30T20:10:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:27.883 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 37 M 2023-10-30T20:10:27.883 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 209 M 2023-10-30T20:10:27.883 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:10:27.994 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 277 kB/s | 32 kB 00:00 2023-10-30T20:10:28.027 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 531 kB/s | 17 kB 00:00 2023-10-30T20:10:28.061 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-10-30T20:10:28.120 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-10-30T20:10:28.186 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.1 MB/s | 205 kB 00:00 2023-10-30T20:10:28.245 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2023-10-30T20:10:28.320 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2023-10-30T20:10:28.346 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-10-30T20:10:28.413 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 12 MB/s | 803 kB 00:00 2023-10-30T20:10:28.571 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 63 MB/s | 16 MB 00:00 2023-10-30T20:10:28.596 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2023-10-30T20:10:28.697 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 9.0 MB/s | 7.4 MB 00:00 2023-10-30T20:10:28.699 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:28.700 INFO:teuthology.orchestra.run.smithi183.stdout:Total 45 MB/s | 37 MB 00:00 2023-10-30T20:10:28.700 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:10:28.722 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:10:28.722 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:10:29.156 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:10:29.157 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:10:29.570 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 33/33 2023-10-30T20:10:29.570 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/33 2023-10-30T20:10:29.570 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/33 2023-10-30T20:10:29.571 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 3/33 2023-10-30T20:10:29.571 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 4/33 2023-10-30T20:10:29.571 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/33 2023-10-30T20:10:29.571 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9. 6/33 2023-10-30T20:10:29.571 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-30T20:10:29.572 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-30T20:10:29.572 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-30T20:10:29.572 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-30T20:10:29.572 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T20:10:29.572 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T20:10:29.573 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-30T20:10:29.573 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-10-30T20:10:29.573 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-30T20:10:29.573 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-30T20:10:29.573 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-30T20:10:29.574 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-30T20:10:29.574 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-30T20:10:29.574 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-30T20:10:29.574 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-30T20:10:29.574 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-30T20:10:29.575 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-30T20:10:29.575 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-30T20:10:29.575 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-30T20:10:29.576 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-30T20:10:29.576 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T20:10:29.576 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-30T20:10:29.577 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-30T20:10:29.577 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-30T20:10:29.577 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-30T20:10:29.577 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-30T20:10:29.852 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:10:29.964 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-30T20:10:29.992 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-30T20:10:30.086 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-30T20:10:30.129 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-30T20:10:30.319 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T20:10:30.345 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T20:10:30.430 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-30T20:10:30.473 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-30T20:10:30.473 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:30.474 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:10:30.474 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:30.474 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:30.474 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:30.475 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:30.475 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:30.475 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:30.475 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-30T20:10:30.475 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-30T20:10:30.476 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-30T20:10:30.476 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-30T20:10:30.476 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-30T20:10:30.476 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-30T20:10:30.477 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-30T20:10:30.477 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-30T20:10:30.477 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-30T20:10:30.477 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-30T20:10:30.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-30T20:10:30.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-30T20:10:30.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-30T20:10:30.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-30T20:10:30.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-30T20:10:30.479 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply-3.11-14.el9.noarch 2023-10-30T20:10:30.479 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-30T20:10:30.479 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-30T20:10:30.479 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-30T20:10:30.479 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-30T20:10:30.480 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-30T20:10:30.480 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-30T20:10:30.480 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-30T20:10:30.480 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-10-30T20:10:30.480 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-30T20:10:30.481 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-30T20:10:30.481 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-30T20:10:30.481 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:30.481 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:10:30.804 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-base 2023-10-30T20:10:31.076 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-30T20:10:31.190 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T20:10:31.355 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:58 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:10:31.446 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:10:31.495 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:10:31.496 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-10-30T20:10:31.496 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:10:31.572 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install cephadm 2023-10-30T20:10:32.118 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:59 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:10:32.258 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:10:32.259 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:10:32.260 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-10-30T20:10:32.260 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:10:32.260 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:10:32.260 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 270 k 2023-10-30T20:10:32.260 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:32.261 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:10:32.261 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:10:32.261 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:10:32.261 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:32.261 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 270 k 2023-10-30T20:10:32.261 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 265 k 2023-10-30T20:10:32.262 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:10:32.434 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-30T20:10:32.504 INFO:teuthology.orchestra.run.smithi069.stdout:cephadm-18.0.0-6839.gcc36dd85.el9.noarch.rpm 1.1 MB/s | 270 kB 00:00 2023-10-30T20:10:32.505 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:32.505 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.1 MB/s | 270 kB 00:00 2023-10-30T20:10:32.506 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:10:32.510 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:10:32.511 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:10:32.513 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:10:32.514 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:10:32.587 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:10:33.051 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T20:10:33.142 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T20:10:33.464 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-30T20:10:33.494 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 12/12 2023-10-30T20:10:33.522 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T20:10:33.839 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T20:10:33.839 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:33.839 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:10:33.840 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:33.840 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:33.840 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:10:34.034 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-immutable-object-cache 2023-10-30T20:10:34.584 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:01 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:10:34.727 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:10:34.728 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:10:34.728 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:10:34.728 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:10:34.728 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:10:34.729 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 141 k 2023-10-30T20:10:34.729 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:34.729 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:10:34.729 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:10:34.729 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:10:34.729 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:34.730 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 141 k 2023-10-30T20:10:34.730 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 395 k 2023-10-30T20:10:34.730 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:10:35.177 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-immutable-object-cache-18.0.0-6839.gcc36dd 314 kB/s | 141 kB 00:00 2023-10-30T20:10:35.178 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:35.178 INFO:teuthology.orchestra.run.smithi069.stdout:Total 313 kB/s | 141 kB 00:00 2023-10-30T20:10:35.178 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:10:35.185 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:10:35.186 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:10:35.238 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:10:35.238 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:10:35.416 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:10:35.478 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T20:10:35.503 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T20:10:35.503 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:10:35.503 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-30T20:10:35.504 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:35.987 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 12/12 2023-10-30T20:10:35.988 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 1/12 2023-10-30T20:10:35.988 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-30T20:10:35.988 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-30T20:10:35.988 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-30T20:10:35.989 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T20:10:35.989 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T20:10:35.989 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-30T20:10:35.989 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-30T20:10:35.989 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T20:10:35.990 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-30T20:10:35.990 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-30T20:10:36.439 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-30T20:10:36.440 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:36.440 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:10:36.440 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:36.440 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-30T20:10:36.441 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-30T20:10:36.441 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-30T20:10:36.441 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-30T20:10:36.441 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-30T20:10:36.442 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-30T20:10:36.442 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-30T20:10:36.442 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-30T20:10:36.442 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-30T20:10:36.442 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-30T20:10:36.443 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-30T20:10:36.443 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:36.443 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:36.493 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T20:10:36.494 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:36.494 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:10:36.494 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:36.494 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:36.495 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:10:36.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr 2023-10-30T20:10:36.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-rook 2023-10-30T20:10:37.338 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:13 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:37.341 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:04 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:10:37.433 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:10:37.482 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:10:37.482 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:37.483 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-10-30T20:10:37.483 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:10:37.483 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2023-10-30T20:10:37.484 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2023-10-30T20:10:37.484 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2023-10-30T20:10:37.484 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:10:37.484 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 48 k 2023-10-30T20:10:37.485 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2023-10-30T20:10:37.485 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-30T20:10:37.485 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-30T20:10:37.485 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-30T20:10:37.485 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-30T20:10:37.485 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-30T20:10:37.486 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-30T20:10:37.486 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-30T20:10:37.486 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-30T20:10:37.486 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-30T20:10:37.486 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-30T20:10:37.487 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-30T20:10:37.487 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-30T20:10:37.487 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:37.487 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:10:37.487 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2023-10-30T20:10:37.489 INFO:teuthology.orchestra.run.smithi183.stdout:Install 13 Packages 2023-10-30T20:10:37.489 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:37.490 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.0 M 2023-10-30T20:10:37.490 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 26 M 2023-10-30T20:10:37.490 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:10:37.553 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-dashboard 2023-10-30T20:10:37.885 INFO:teuthology.orchestra.run.smithi183.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 202 kB/s | 24 kB 00:00 2023-10-30T20:10:37.910 INFO:teuthology.orchestra.run.smithi183.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2023-10-30T20:10:37.952 INFO:teuthology.orchestra.run.smithi183.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-10-30T20:10:37.977 INFO:teuthology.orchestra.run.smithi183.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-10-30T20:10:38.003 INFO:teuthology.orchestra.run.smithi183.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-30T20:10:38.028 INFO:teuthology.orchestra.run.smithi183.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2023-10-30T20:10:38.053 INFO:teuthology.orchestra.run.smithi183.stdout:(7/13): ceph-mgr-rook-18.0.0-6839.gcc36dd85.el9 165 kB/s | 48 kB 00:00 2023-10-30T20:10:38.093 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:05 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:10:38.235 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:10:38.236 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:10:38.236 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-10-30T20:10:38.236 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:10:38.237 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:10:38.237 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 1.8 M 2023-10-30T20:10:38.237 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-10-30T20:10:38.237 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 23 k 2023-10-30T20:10:38.238 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 15 k 2023-10-30T20:10:38.238 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-30T20:10:38.238 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-30T20:10:38.238 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-30T20:10:38.238 INFO:teuthology.orchestra.run.smithi069.stdout:Installing weak dependencies: 2023-10-30T20:10:38.239 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-30T20:10:38.239 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:38.239 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:10:38.239 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:10:38.239 INFO:teuthology.orchestra.run.smithi069.stdout:Install 7 Packages 2023-10-30T20:10:38.239 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:38.240 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 2.1 M 2023-10-30T20:10:38.240 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 8.6 M 2023-10-30T20:10:38.241 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:10:38.329 INFO:teuthology.orchestra.run.smithi183.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2023-10-30T20:10:38.362 INFO:teuthology.orchestra.run.smithi183.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 90 kB/s | 32 kB 00:00 2023-10-30T20:10:38.637 INFO:teuthology.orchestra.run.smithi183.stdout:(10/13): python3-google-auth-2.23.3-1.el9.noarc 337 kB/s | 197 kB 00:00 2023-10-30T20:10:38.663 INFO:teuthology.orchestra.run.smithi183.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 197 kB/s | 59 kB 00:00 2023-10-30T20:10:38.805 INFO:teuthology.orchestra.run.smithi069.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6839.gcc3 65 kB/s | 23 kB 00:00 2023-10-30T20:10:38.839 INFO:teuthology.orchestra.run.smithi069.stdout:(2/7): ceph-prometheus-alerts-18.0.0-6839.gcc36 39 kB/s | 15 kB 00:00 2023-10-30T20:10:38.854 INFO:teuthology.orchestra.run.smithi183.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 413 kB/s | 90 kB 00:00 2023-10-30T20:10:38.898 INFO:teuthology.orchestra.run.smithi069.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 98 kB/s | 9.0 kB 00:00 2023-10-30T20:10:38.988 INFO:teuthology.orchestra.run.smithi183.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2023-10-30T20:10:38.990 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:38.991 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-10-30T20:10:38.991 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:10:39.006 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:10:39.006 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:10:39.015 INFO:teuthology.orchestra.run.smithi069.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 263 kB/s | 31 kB 00:00 2023-10-30T20:10:39.040 INFO:teuthology.orchestra.run.smithi069.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 204 kB/s | 41 kB 00:00 2023-10-30T20:10:39.109 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:10:39.110 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:10:39.165 INFO:teuthology.orchestra.run.smithi069.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2023-10-30T20:10:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:10:39.665 INFO:teuthology.orchestra.run.smithi069.stdout:(7/7): ceph-mgr-dashboard-18.0.0-6839.gcc36dd85 1.5 MB/s | 1.8 MB 00:01 2023-10-30T20:10:39.667 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:39.667 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.4 MB/s | 2.1 MB 00:01 2023-10-30T20:10:39.668 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:10:39.676 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:10:39.676 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:10:39.749 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:10:39.749 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:10:39.793 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-30T20:10:39.847 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-30T20:10:39.911 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-30T20:10:39.948 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-30T20:10:39.990 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-30T20:10:40.042 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:10:40.084 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-30T20:10:40.108 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-30T20:10:40.142 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-30T20:10:40.188 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-30T20:10:40.194 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-10-30T20:10:40.221 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-30T20:10:40.243 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-30T20:10:40.249 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.n 5/7 2023-10-30T20:10:40.485 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9. 6/7 2023-10-30T20:10:40.519 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 7/7 2023-10-30T20:10:40.959 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-30T20:10:41.007 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-30T20:10:41.041 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-30T20:10:41.061 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 7/7 2023-10-30T20:10:41.061 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9. 1/7 2023-10-30T20:10:41.061 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 2/7 2023-10-30T20:10:41.061 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.n 3/7 2023-10-30T20:10:41.062 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-30T20:10:41.062 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-30T20:10:41.062 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-30T20:10:41.086 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-30T20:10:41.111 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 13/13 2023-10-30T20:10:41.373 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-30T20:10:41.373 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:10:41.373 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:41.374 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:41.374 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:41.374 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-30T20:10:41.374 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-30T20:10:41.374 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-30T20:10:41.375 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-30T20:10:41.375 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:10:41.567 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-30T20:10:41.901 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 13/13 2023-10-30T20:10:41.901 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 1/13 2023-10-30T20:10:41.901 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-30T20:10:41.902 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-30T20:10:41.902 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-30T20:10:41.902 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-30T20:10:41.902 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-30T20:10:41.902 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-30T20:10:41.903 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-30T20:10:41.903 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-30T20:10:41.903 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-10-30T20:10:41.903 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-30T20:10:41.903 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-30T20:10:42.110 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:09 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:10:42.251 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:10:42.252 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2023-10-30T20:10:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-10-30T20:10:42.252 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2023-10-30T20:10:42.253 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:10:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 7.4 M 2023-10-30T20:10:42.253 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-10-30T20:10:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-30T20:10:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-30T20:10:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-30T20:10:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-30T20:10:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-30T20:10:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-30T20:10:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-30T20:10:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-30T20:10:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-30T20:10:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-30T20:10:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-30T20:10:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:42.256 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:10:42.256 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2023-10-30T20:10:42.256 INFO:teuthology.orchestra.run.smithi069.stdout:Install 12 Packages 2023-10-30T20:10:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:42.257 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 37 M 2023-10-30T20:10:42.257 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 209 M 2023-10-30T20:10:42.257 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:10:42.281 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-30T20:10:42.282 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:42.282 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:10:42.282 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:42.282 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-30T20:10:42.282 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-30T20:10:42.283 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-30T20:10:42.283 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-30T20:10:42.283 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-30T20:10:42.283 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-30T20:10:42.283 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-30T20:10:42.283 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-30T20:10:42.284 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-30T20:10:42.284 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-30T20:10:42.284 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-30T20:10:42.284 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-30T20:10:42.284 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:42.285 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:42.389 INFO:teuthology.orchestra.run.smithi069.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 236 kB/s | 32 kB 00:00 2023-10-30T20:10:42.422 INFO:teuthology.orchestra.run.smithi069.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 533 kB/s | 17 kB 00:00 2023-10-30T20:10:42.456 INFO:teuthology.orchestra.run.smithi069.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-10-30T20:10:42.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-cephadm 2023-10-30T20:10:42.514 INFO:teuthology.orchestra.run.smithi069.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-10-30T20:10:42.640 INFO:teuthology.orchestra.run.smithi069.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2023-10-30T20:10:42.665 INFO:teuthology.orchestra.run.smithi069.stdout:(6/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 205 kB 00:00 2023-10-30T20:10:42.740 INFO:teuthology.orchestra.run.smithi069.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 48 MB/s | 4.8 MB 00:00 2023-10-30T20:10:42.765 INFO:teuthology.orchestra.run.smithi069.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-10-30T20:10:42.832 INFO:teuthology.orchestra.run.smithi069.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 12 MB/s | 803 kB 00:00 2023-10-30T20:10:43.008 INFO:teuthology.orchestra.run.smithi069.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 59 MB/s | 16 MB 00:00 2023-10-30T20:10:43.033 INFO:teuthology.orchestra.run.smithi069.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 987 kB/s | 195 kB 00:00 2023-10-30T20:10:43.089 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:19 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:43.216 INFO:teuthology.orchestra.run.smithi069.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 7.7 MB/s | 7.4 MB 00:00 2023-10-30T20:10:43.219 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:43.220 INFO:teuthology.orchestra.run.smithi069.stdout:Total 38 MB/s | 37 MB 00:00 2023-10-30T20:10:43.220 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:10:43.236 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:43.237 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2023-10-30T20:10:43.238 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2023-10-30T20:10:43.238 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2023-10-30T20:10:43.238 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:10:43.238 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 132 k 2023-10-30T20:10:43.238 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2023-10-30T20:10:43.239 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-30T20:10:43.239 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-30T20:10:43.239 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-30T20:10:43.239 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-30T20:10:43.239 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-30T20:10:43.240 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:43.240 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:10:43.240 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2023-10-30T20:10:43.240 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2023-10-30T20:10:43.240 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:43.241 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 6.8 M 2023-10-30T20:10:43.241 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 33 M 2023-10-30T20:10:43.241 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:10:43.242 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:10:43.242 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:10:43.651 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-10-30T20:10:43.667 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:10:43.668 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:10:43.767 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2023-10-30T20:10:43.984 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 266 kB/s | 58 kB 00:00 2023-10-30T20:10:44.010 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): ceph-mgr-cephadm-18.0.0-6839.gcc36dd85.e 258 kB/s | 132 kB 00:00 2023-10-30T20:10:44.068 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 610 kB/s | 51 kB 00:00 2023-10-30T20:10:44.127 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2023-10-30T20:10:44.128 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:44.129 INFO:teuthology.orchestra.run.smithi183.stdout:Total 7.6 MB/s | 6.8 MB 00:00 2023-10-30T20:10:44.129 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:10:44.138 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:10:44.138 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:10:44.227 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:10:44.228 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:10:44.362 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:10:44.441 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-30T20:10:44.471 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-30T20:10:44.526 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:10:44.572 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-30T20:10:44.624 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-30T20:10:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-30T20:10:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-30T20:10:44.804 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T20:10:44.831 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T20:10:44.915 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-30T20:10:45.263 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-30T20:10:45.347 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-30T20:10:45.408 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-30T20:10:45.433 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 6/6 2023-10-30T20:10:45.563 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-30T20:10:45.677 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T20:10:46.269 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 6/6 2023-10-30T20:10:46.269 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/6 2023-10-30T20:10:46.269 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-30T20:10:46.270 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-30T20:10:46.270 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-30T20:10:46.270 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-30T20:10:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-30T20:10:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:46.590 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:10:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-30T20:10:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-30T20:10:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-30T20:10:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-30T20:10:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-30T20:10:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:46.592 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:46.864 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-fuse 2023-10-30T20:10:46.914 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-30T20:10:47.422 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:23 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:47.564 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:47.565 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:47.565 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2023-10-30T20:10:47.565 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:47.565 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:10:47.566 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 855 k 2023-10-30T20:10:47.566 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2023-10-30T20:10:47.566 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-30T20:10:47.566 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:47.566 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:10:47.567 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:47.567 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2023-10-30T20:10:47.567 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:47.567 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 934 k 2023-10-30T20:10:47.568 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.7 M 2023-10-30T20:10:47.568 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:10:47.679 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 698 kB/s | 79 kB 00:00 2023-10-30T20:10:47.920 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-fuse-18.0.0-6839.gcc36dd85.el9.x86_ 2.4 MB/s | 855 kB 00:00 2023-10-30T20:10:47.921 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:47.922 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.6 MB/s | 934 kB 00:00 2023-10-30T20:10:47.922 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:10:47.933 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-30T20:10:47.942 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:10:47.943 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:10:47.964 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 12/12 2023-10-30T20:10:48.022 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:10:48.022 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:10:48.221 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:10:48.361 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-30T20:10:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T20:10:49.184 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T20:10:49.184 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/2 2023-10-30T20:10:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-30T20:10:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:49.505 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:10:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-30T20:10:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:49.505 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:49.793 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-volume 2023-10-30T20:10:50.348 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:26 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:50.484 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 12/12 2023-10-30T20:10:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 1/12 2023-10-30T20:10:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-30T20:10:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-30T20:10:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-30T20:10:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T20:10:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T20:10:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-30T20:10:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-30T20:10:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T20:10:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-30T20:10:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-30T20:10:50.489 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:50.490 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:50.490 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2023-10-30T20:10:50.490 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:50.490 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:10:50.491 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 262 k 2023-10-30T20:10:50.491 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:50.491 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:10:50.491 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:50.491 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:10:50.492 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:50.492 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 262 k 2023-10-30T20:10:50.492 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.2 M 2023-10-30T20:10:50.492 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:10:50.916 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-30T20:10:50.917 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:50.917 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:10:50.917 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:50.917 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-30T20:10:50.917 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-30T20:10:50.918 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-30T20:10:50.918 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-30T20:10:50.918 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-30T20:10:50.918 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-30T20:10:50.918 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-30T20:10:50.919 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-30T20:10:50.919 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-30T20:10:50.919 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-30T20:10:50.919 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-30T20:10:50.919 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:50.920 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:10:51.075 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-volume-18.0.0-6839.gcc36dd85.el9.noarch.rp 449 kB/s | 262 kB 00:00 2023-10-30T20:10:51.075 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:51.076 INFO:teuthology.orchestra.run.smithi183.stdout:Total 448 kB/s | 262 kB 00:00 2023-10-30T20:10:51.076 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:10:51.083 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:10:51.083 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:10:51.122 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:10:51.122 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:10:51.290 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-rook 2023-10-30T20:10:51.339 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:10:51.393 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T20:10:51.408 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T20:10:51.408 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:10:51.409 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-30T20:10:51.409 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:51.843 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:18 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:10:51.986 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:10:51.987 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2023-10-30T20:10:51.987 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-10-30T20:10:51.987 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2023-10-30T20:10:51.987 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:10:51.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 48 k 2023-10-30T20:10:51.988 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-10-30T20:10:51.988 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-30T20:10:51.988 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-30T20:10:51.988 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-30T20:10:51.988 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-30T20:10:51.989 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-30T20:10:51.989 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-30T20:10:51.989 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-30T20:10:51.989 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-30T20:10:51.989 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-30T20:10:51.990 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-30T20:10:51.990 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-30T20:10:51.990 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-30T20:10:51.990 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:51.990 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:10:51.991 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2023-10-30T20:10:51.991 INFO:teuthology.orchestra.run.smithi069.stdout:Install 13 Packages 2023-10-30T20:10:51.991 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:51.992 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 2.0 M 2023-10-30T20:10:51.992 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 26 M 2023-10-30T20:10:51.992 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:10:52.446 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T20:10:52.446 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:52.446 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:10:52.447 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:52.447 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:52.447 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:52.472 INFO:teuthology.orchestra.run.smithi069.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 203 kB/s | 24 kB 00:00 2023-10-30T20:10:52.497 INFO:teuthology.orchestra.run.smithi069.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2023-10-30T20:10:52.540 INFO:teuthology.orchestra.run.smithi069.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-10-30T20:10:52.565 INFO:teuthology.orchestra.run.smithi069.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2023-10-30T20:10:52.590 INFO:teuthology.orchestra.run.smithi069.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-30T20:10:52.615 INFO:teuthology.orchestra.run.smithi069.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2023-10-30T20:10:52.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados-devel 2023-10-30T20:10:52.749 INFO:teuthology.orchestra.run.smithi069.stdout:(7/13): ceph-mgr-rook-18.0.0-6839.gcc36dd85.el9 120 kB/s | 48 kB 00:00 2023-10-30T20:10:52.891 INFO:teuthology.orchestra.run.smithi069.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2023-10-30T20:10:52.916 INFO:teuthology.orchestra.run.smithi069.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2023-10-30T20:10:53.108 INFO:teuthology.orchestra.run.smithi069.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2023-10-30T20:10:53.225 INFO:teuthology.orchestra.run.smithi069.stdout:(11/13): python3-google-auth-2.23.3-1.el9.noarc 414 kB/s | 197 kB 00:00 2023-10-30T20:10:53.300 INFO:teuthology.orchestra.run.smithi069.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 466 kB/s | 90 kB 00:00 2023-10-30T20:10:53.307 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:29 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:53.425 INFO:teuthology.orchestra.run.smithi069.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2023-10-30T20:10:53.428 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:53.428 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-10-30T20:10:53.428 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:10:53.442 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:10:53.442 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:10:53.450 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:53.451 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:53.451 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:10:53.451 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:53.451 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:10:53.451 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 127 k 2023-10-30T20:10:53.452 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:53.452 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:10:53.452 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:53.452 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:10:53.452 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:53.453 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 127 k 2023-10-30T20:10:53.453 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 456 k 2023-10-30T20:10:53.453 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:10:53.542 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:10:53.542 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:10:53.930 INFO:teuthology.orchestra.run.smithi183.stdout:librados-devel-18.0.0-6839.gcc36dd85.el9.x86_64 265 kB/s | 127 kB 00:00 2023-10-30T20:10:53.931 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:53.931 INFO:teuthology.orchestra.run.smithi183.stdout:Total 264 kB/s | 127 kB 00:00 2023-10-30T20:10:53.931 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:10:53.937 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:10:53.937 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:10:53.992 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:10:53.993 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:10:54.068 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:10:54.136 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:10:54.240 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-30T20:10:54.244 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:10:54.293 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-30T20:10:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-30T20:10:54.394 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-30T20:10:54.435 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-30T20:10:54.528 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-30T20:10:54.638 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-10-30T20:10:54.695 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-30T20:10:54.706 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:10:55.027 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:10:55.027 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:55.027 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:10:55.027 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:55.028 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:55.028 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:55.264 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs2 2023-10-30T20:10:55.425 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-30T20:10:55.467 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-30T20:10:55.518 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-30T20:10:55.565 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-30T20:10:55.588 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 13/13 2023-10-30T20:10:55.818 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:31 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:55.912 INFO:teuthology.orchestra.run.smithi183.stdout:Package libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:10:55.961 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:55.962 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2023-10-30T20:10:55.963 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:56.017 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs-devel 2023-10-30T20:10:56.449 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 13/13 2023-10-30T20:10:56.450 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 1/13 2023-10-30T20:10:56.450 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-30T20:10:56.450 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-30T20:10:56.450 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-30T20:10:56.450 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-30T20:10:56.451 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-30T20:10:56.451 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-30T20:10:56.451 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-30T20:10:56.451 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-30T20:10:56.451 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-10-30T20:10:56.452 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-30T20:10:56.452 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-30T20:10:56.563 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:32 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:56.706 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:56.706 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:10:56.707 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:56.707 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:10:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 31 k 2023-10-30T20:10:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:56.708 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:10:56.708 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:10:56.708 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:10:56.708 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:56.708 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 31 k 2023-10-30T20:10:56.709 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 139 k 2023-10-30T20:10:56.709 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:10:56.807 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-30T20:10:56.807 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:56.807 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:10:56.808 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:10:56.808 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-30T20:10:56.808 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-30T20:10:56.808 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-30T20:10:56.808 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-30T20:10:56.809 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-30T20:10:56.809 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-30T20:10:56.809 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-30T20:10:56.809 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-30T20:10:56.809 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-30T20:10:56.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-30T20:10:56.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-30T20:10:56.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-30T20:10:56.810 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:56.810 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:10:56.875 INFO:teuthology.orchestra.run.smithi183.stdout:libcephfs-devel-18.0.0-6839.gcc36dd85.el9.x86_6 186 kB/s | 31 kB 00:00 2023-10-30T20:10:56.876 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:56.876 INFO:teuthology.orchestra.run.smithi183.stdout:Total 184 kB/s | 31 kB 00:00 2023-10-30T20:10:56.876 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:10:56.879 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:10:56.879 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:10:56.919 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:10:56.920 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:10:57.009 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:10:57.067 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-cephadm 2023-10-30T20:10:57.105 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:10:57.470 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:10:57.612 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:24 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:10:57.752 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:10:57.753 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:57.753 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:10:57.753 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:10:57.753 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:10:57.753 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:57.754 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:10:57.755 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2023-10-30T20:10:57.755 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-10-30T20:10:57.755 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2023-10-30T20:10:57.755 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:10:57.756 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 132 k 2023-10-30T20:10:57.756 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-10-30T20:10:57.756 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-30T20:10:57.756 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-30T20:10:57.756 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-30T20:10:57.756 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-30T20:10:57.757 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-30T20:10:57.757 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:57.757 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:10:57.757 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2023-10-30T20:10:57.757 INFO:teuthology.orchestra.run.smithi069.stdout:Install 6 Packages 2023-10-30T20:10:57.758 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:10:57.758 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 6.8 M 2023-10-30T20:10:57.758 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 33 M 2023-10-30T20:10:57.758 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:10:57.923 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados2 2023-10-30T20:10:58.269 INFO:teuthology.orchestra.run.smithi069.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-10-30T20:10:58.402 INFO:teuthology.orchestra.run.smithi069.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2023-10-30T20:10:58.428 INFO:teuthology.orchestra.run.smithi069.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6839.gcc36dd85.e 432 kB/s | 132 kB 00:00 2023-10-30T20:10:58.471 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:34 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:58.561 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:10:58.611 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:58.612 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2023-10-30T20:10:58.612 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:58.677 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librbd1 2023-10-30T20:10:58.711 INFO:teuthology.orchestra.run.smithi069.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 187 kB/s | 58 kB 00:00 2023-10-30T20:10:58.736 INFO:teuthology.orchestra.run.smithi069.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 165 kB/s | 51 kB 00:00 2023-10-30T20:10:58.870 INFO:teuthology.orchestra.run.smithi069.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 912 kB/s | 548 kB 00:00 2023-10-30T20:10:58.872 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:10:58.872 INFO:teuthology.orchestra.run.smithi069.stdout:Total 6.1 MB/s | 6.8 MB 00:01 2023-10-30T20:10:58.873 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:10:58.882 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:10:58.882 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:10:58.970 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:10:58.971 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:10:59.227 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:35 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:10:59.294 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:10:59.318 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:10:59.367 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:10:59.368 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2023-10-30T20:10:59.369 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:10:59.436 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rados 2023-10-30T20:10:59.458 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-30T20:10:59.504 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-30T20:10:59.980 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:35 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:11:00.010 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-30T20:11:00.071 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:11:00.090 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-30T20:11:00.124 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:11:00.125 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2023-10-30T20:11:00.125 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:11:00.151 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-30T20:11:00.175 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 6/6 2023-10-30T20:11:00.189 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rgw 2023-10-30T20:11:00.735 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:36 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:11:00.826 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:11:00.878 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:11:00.878 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2023-10-30T20:11:00.879 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:11:00.939 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-cephfs 2023-10-30T20:11:00.963 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 6/6 2023-10-30T20:11:00.963 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/6 2023-10-30T20:11:00.963 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-30T20:11:00.964 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-30T20:11:00.964 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-30T20:11:00.964 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-30T20:11:01.271 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-30T20:11:01.271 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:01.271 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:11:01.271 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:11:01.272 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-30T20:11:01.272 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-30T20:11:01.272 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-30T20:11:01.272 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-30T20:11:01.272 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-30T20:11:01.273 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:01.273 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:01.486 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:37 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:11:01.532 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-fuse 2023-10-30T20:11:01.577 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:11:01.628 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:11:01.642 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2023-10-30T20:11:01.643 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:11:01.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rbd 2023-10-30T20:11:02.082 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:29 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:02.224 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:02.225 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:02.225 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-10-30T20:11:02.225 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:02.226 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:11:02.226 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 855 k 2023-10-30T20:11:02.226 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-10-30T20:11:02.226 INFO:teuthology.orchestra.run.smithi069.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-30T20:11:02.226 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:02.227 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:11:02.227 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:02.227 INFO:teuthology.orchestra.run.smithi069.stdout:Install 2 Packages 2023-10-30T20:11:02.227 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:02.228 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 934 k 2023-10-30T20:11:02.228 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.7 M 2023-10-30T20:11:02.228 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:11:02.241 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:38 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:11:02.332 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:11:02.338 INFO:teuthology.orchestra.run.smithi069.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 704 kB/s | 79 kB 00:00 2023-10-30T20:11:02.381 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:11:02.382 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2023-10-30T20:11:02.382 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:11:02.444 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-fuse 2023-10-30T20:11:02.547 INFO:teuthology.orchestra.run.smithi069.stdout:(2/2): ceph-fuse-18.0.0-6839.gcc36dd85.el9.x86_ 2.6 MB/s | 855 kB 00:00 2023-10-30T20:11:02.548 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:11:02.548 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.8 MB/s | 934 kB 00:00 2023-10-30T20:11:02.549 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:11:02.568 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:11:02.569 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:11:02.646 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:11:02.647 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:11:02.849 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:11:02.978 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-30T20:11:02.995 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:38 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:11:03.046 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T20:11:03.137 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:11:03.138 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:03.138 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2023-10-30T20:11:03.138 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:03.138 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:11:03.138 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 85 k 2023-10-30T20:11:03.139 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:03.139 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:11:03.139 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:03.139 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:11:03.139 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:03.139 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 85 k 2023-10-30T20:11:03.140 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 227 k 2023-10-30T20:11:03.140 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:11:03.694 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T20:11:03.695 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/2 2023-10-30T20:11:03.695 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-fuse-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 153 kB/s | 85 kB 00:00 2023-10-30T20:11:03.696 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:11:03.696 INFO:teuthology.orchestra.run.smithi183.stdout:Total 152 kB/s | 85 kB 00:00 2023-10-30T20:11:03.696 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:11:03.703 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:11:03.703 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:11:03.759 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:11:03.759 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:11:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:11:04.001 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:04.019 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-30T20:11:04.020 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:04.020 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:11:04.020 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-30T20:11:04.020 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:04.020 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:04.270 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-volume 2023-10-30T20:11:04.402 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:04.701 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:04.701 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:04.702 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:11:04.702 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:11:04.702 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:04.702 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:11:04.824 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:31 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:04.923 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-mirror 2023-10-30T20:11:04.970 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:04.971 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:04.971 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-10-30T20:11:04.971 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:04.971 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:11:04.972 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 262 k 2023-10-30T20:11:04.972 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:04.972 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:11:04.972 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:04.972 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:11:04.973 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:04.973 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 262 k 2023-10-30T20:11:04.973 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.2 M 2023-10-30T20:11:04.973 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:11:05.393 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-volume-18.0.0-6839.gcc36dd85.el9.noarch.rp 622 kB/s | 262 kB 00:00 2023-10-30T20:11:05.394 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:11:05.394 INFO:teuthology.orchestra.run.smithi069.stdout:Total 621 kB/s | 262 kB 00:00 2023-10-30T20:11:05.394 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:11:05.401 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:11:05.402 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:11:05.440 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:11:05.440 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:11:05.471 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:41 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:11:05.616 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:11:05.617 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:05.618 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:11:05.618 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:05.618 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:11:05.618 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.0 M 2023-10-30T20:11:05.618 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:05.619 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:11:05.619 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:05.619 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:11:05.619 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:05.619 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 3.0 M 2023-10-30T20:11:05.619 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 12 M 2023-10-30T20:11:05.620 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:11:05.655 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:11:05.728 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T20:11:05.744 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T20:11:05.744 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:11:05.744 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-30T20:11:05.744 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:06.811 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T20:11:06.811 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:06.812 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:11:06.812 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T20:11:06.812 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:06.812 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:07.153 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados-devel 2023-10-30T20:11:07.172 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-mirror-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 1.9 MB/s | 3.0 MB 00:01 2023-10-30T20:11:07.173 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:11:07.174 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.9 MB/s | 3.0 MB 00:01 2023-10-30T20:11:07.174 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:11:07.183 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:11:07.184 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:11:07.260 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:11:07.260 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:11:07.673 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:11:07.709 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:34 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:07.731 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:07.755 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:07.756 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:11:07.756 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-30T20:11:07.756 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T20:11:07.756 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T20:11:07.757 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:07.851 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:07.852 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:07.852 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:11:07.852 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:07.853 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:11:07.853 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 127 k 2023-10-30T20:11:07.853 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:07.853 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:11:07.853 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:07.853 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:11:07.854 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:07.854 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 127 k 2023-10-30T20:11:07.854 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 456 k 2023-10-30T20:11:07.854 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:11:08.331 INFO:teuthology.orchestra.run.smithi069.stdout:librados-devel-18.0.0-6839.gcc36dd85.el9.x86_64 265 kB/s | 127 kB 00:00 2023-10-30T20:11:08.332 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:11:08.332 INFO:teuthology.orchestra.run.smithi069.stdout:Total 264 kB/s | 127 kB 00:00 2023-10-30T20:11:08.332 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:11:08.337 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:11:08.337 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:11:08.393 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:11:08.393 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:11:08.546 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:11:08.646 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:08.859 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:08.859 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:08.860 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:11:08.860 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:11:08.860 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:08.860 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:11:09.091 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-nbd 2023-10-30T20:11:09.098 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:09.384 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:09.385 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:09.385 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:11:09.385 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:11:09.385 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:09.385 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:09.648 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs2 2023-10-30T20:11:09.661 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:45 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:11:09.804 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:11:09.805 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:09.805 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2023-10-30T20:11:09.806 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:09.806 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:11:09.806 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 168 k 2023-10-30T20:11:09.806 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:09.806 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:11:09.807 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:09.807 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:11:09.807 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:09.807 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 168 k 2023-10-30T20:11:09.807 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 486 k 2023-10-30T20:11:09.808 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:11:10.034 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-nbd-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 741 kB/s | 168 kB 00:00 2023-10-30T20:11:10.034 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:11:10.035 INFO:teuthology.orchestra.run.smithi183.stdout:Total 736 kB/s | 168 kB 00:00 2023-10-30T20:11:10.035 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:11:10.042 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:11:10.042 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:11:10.097 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:11:10.098 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:11:10.205 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:37 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:10.246 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:11:10.299 INFO:teuthology.orchestra.run.smithi069.stdout:Package libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:11:10.351 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:10.352 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-10-30T20:11:10.352 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:10.357 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:10.423 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs-devel 2023-10-30T20:11:10.822 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:10.965 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:37 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:11.106 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:11.107 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:11.107 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:11:11.107 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:11.107 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:11:11.107 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 31 k 2023-10-30T20:11:11.108 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:11.108 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:11:11.108 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:11.108 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:11:11.108 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:11.108 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 31 k 2023-10-30T20:11:11.109 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 139 k 2023-10-30T20:11:11.109 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:11:11.116 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:11.116 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:11.116 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:11:11.116 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:11:11.116 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:11.117 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:11:11.286 INFO:teuthology.orchestra.run.smithi069.stdout:libcephfs-devel-18.0.0-6839.gcc36dd85.el9.x86_6 174 kB/s | 31 kB 00:00 2023-10-30T20:11:11.287 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:11:11.287 INFO:teuthology.orchestra.run.smithi069.stdout:Total 173 kB/s | 31 kB 00:00 2023-10-30T20:11:11.287 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:11:11.290 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:11:11.290 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:11:11.318 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:11:11.319 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:11:11.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-base-debuginfo 2023-10-30T20:11:11.413 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:11:11.516 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:11.902 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:47 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:11:12.055 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:11:12.056 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:12.056 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:11:12.056 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:12.057 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:11:12.057 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 74 M 2023-10-30T20:11:12.057 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2023-10-30T20:11:12.057 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 20 M 2023-10-30T20:11:12.057 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:12.058 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:11:12.058 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:12.058 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2023-10-30T20:11:12.058 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:12.059 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 94 M 2023-10-30T20:11:12.059 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 511 M 2023-10-30T20:11:12.059 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:11:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:12.158 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:11:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:11:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:12.159 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:12.340 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados2 2023-10-30T20:11:12.886 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:39 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:12.977 INFO:teuthology.orchestra.run.smithi069.stdout:Package librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:11:13.028 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:13.029 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-10-30T20:11:13.029 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:13.095 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librbd1 2023-10-30T20:11:13.562 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): ceph-debuginfo-18.0.0-6839.gcc36dd85.el9 13 MB/s | 20 MB 00:01 2023-10-30T20:11:13.642 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:40 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:13.733 INFO:teuthology.orchestra.run.smithi069.stdout:Package librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:11:13.783 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:13.784 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-10-30T20:11:13.784 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:13.853 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rados 2023-10-30T20:11:14.394 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:41 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:14.485 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:11:14.534 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:14.535 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-10-30T20:11:14.536 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:14.614 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rgw 2023-10-30T20:11:15.157 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:42 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:15.247 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:11:15.296 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:15.298 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-10-30T20:11:15.298 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:15.370 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-cephfs 2023-10-30T20:11:15.910 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:42 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:16.001 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:11:16.051 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:16.052 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-10-30T20:11:16.053 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:16.130 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rbd 2023-10-30T20:11:16.675 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:43 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:16.767 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:11:16.803 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-base-debuginfo-18.0.0-6839.gcc36dd8 16 MB/s | 74 MB 00:04 2023-10-30T20:11:16.806 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:11:16.806 INFO:teuthology.orchestra.run.smithi183.stdout:Total 20 MB/s | 94 MB 00:04 2023-10-30T20:11:16.806 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:11:16.810 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:11:16.811 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:11:16.816 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:16.817 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-10-30T20:11:16.818 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:16.897 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-fuse 2023-10-30T20:11:17.424 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:11:17.424 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:11:17.445 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:44 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:17.587 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2023-10-30T20:11:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:11:17.587 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 85 k 2023-10-30T20:11:17.587 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:11:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:11:17.588 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 85 k 2023-10-30T20:11:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 227 k 2023-10-30T20:11:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:11:17.889 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-fuse-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 281 kB/s | 85 kB 00:00 2023-10-30T20:11:17.890 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:11:17.890 INFO:teuthology.orchestra.run.smithi069.stdout:Total 280 kB/s | 85 kB 00:00 2023-10-30T20:11:17.890 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:11:17.896 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:11:17.897 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:11:17.950 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:11:17.951 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:11:18.089 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:11:18.178 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:18.543 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:18.867 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:18.867 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:18.868 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:11:18.868 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:11:18.868 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:18.868 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:19.097 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-mirror 2023-10-30T20:11:19.652 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:46 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:19.797 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:19.798 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:19.798 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:11:19.798 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:19.799 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:11:19.799 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.0 M 2023-10-30T20:11:19.799 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:19.799 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:11:19.799 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:19.799 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:11:19.800 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:19.800 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 3.0 M 2023-10-30T20:11:19.800 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 12 M 2023-10-30T20:11:19.800 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:11:20.294 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:11:20.400 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-mirror-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2023-10-30T20:11:20.401 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:11:20.401 INFO:teuthology.orchestra.run.smithi069.stdout:Total 5.0 MB/s | 3.0 MB 00:00 2023-10-30T20:11:20.402 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:11:20.411 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:11:20.411 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:11:20.483 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:11:20.484 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:11:20.907 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:11:20.976 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:20.999 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:20.999 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T20:11:21.000 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-30T20:11:21.000 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T20:11:21.000 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T20:11:21.000 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:22.150 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:22.150 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:22.150 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:11:22.150 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:11:22.151 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:22.151 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:22.407 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-nbd 2023-10-30T20:11:22.967 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:49 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:23.108 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:23.109 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:23.109 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2023-10-30T20:11:23.109 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:23.109 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:11:23.110 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 168 k 2023-10-30T20:11:23.110 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:23.110 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:11:23.110 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:23.110 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:11:23.111 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:23.111 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 168 k 2023-10-30T20:11:23.111 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 486 k 2023-10-30T20:11:23.111 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:11:23.662 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-nbd-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 305 kB/s | 168 kB 00:00 2023-10-30T20:11:23.663 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:11:23.663 INFO:teuthology.orchestra.run.smithi069.stdout:Total 304 kB/s | 168 kB 00:00 2023-10-30T20:11:23.663 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:11:23.671 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:11:23.671 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:11:23.727 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:11:23.727 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:11:23.881 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:11:23.977 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:24.379 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:24.700 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:11:24.701 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:24.701 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:11:24.701 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:11:24.701 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:24.701 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:24.934 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-base-debuginfo 2023-10-30T20:11:25.496 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:52 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:25.649 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:25.650 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:25.650 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:11:25.650 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:25.650 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:11:25.651 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 74 M 2023-10-30T20:11:25.651 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-10-30T20:11:25.651 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 20 M 2023-10-30T20:11:25.651 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:25.651 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:11:25.652 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:25.652 INFO:teuthology.orchestra.run.smithi069.stdout:Install 2 Packages 2023-10-30T20:11:25.652 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:25.652 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 94 M 2023-10-30T20:11:25.652 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 511 M 2023-10-30T20:11:25.653 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:11:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/2 2023-10-30T20:11:28.061 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 2/2 2023-10-30T20:11:29.806 INFO:teuthology.orchestra.run.smithi069.stdout:(1/2): ceph-debuginfo-18.0.0-6839.gcc36dd85.el9 4.9 MB/s | 20 MB 00:04 2023-10-30T20:11:31.452 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 2/2 2023-10-30T20:11:31.452 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/2 2023-10-30T20:11:31.737 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T20:11:31.737 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:31.738 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:11:31.738 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:11:31.738 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:11:31.738 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:31.739 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:11:31.927 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-common-debuginfo 2023-10-30T20:11:32.472 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:08 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:11:32.616 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:11:32.616 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:32.616 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:11:32.617 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:32.617 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:11:32.617 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 410 M 2023-10-30T20:11:32.617 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:32.618 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:11:32.618 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:11:32.618 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:11:32.618 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:11:32.618 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 410 M 2023-10-30T20:11:32.619 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.7 G 2023-10-30T20:11:32.619 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:11:37.505 INFO:teuthology.orchestra.run.smithi069.stdout:(2/2): ceph-base-debuginfo-18.0.0-6839.gcc36dd8 6.2 MB/s | 74 MB 00:11 2023-10-30T20:11:37.508 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:11:37.508 INFO:teuthology.orchestra.run.smithi069.stdout:Total 7.9 MB/s | 94 MB 00:11 2023-10-30T20:11:37.508 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:11:37.514 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:11:37.514 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:11:38.129 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:11:38.130 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:11:41.016 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:11:48.695 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/2 2023-10-30T20:11:48.782 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 2/2 2023-10-30T20:11:52.123 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 2/2 2023-10-30T20:11:52.123 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/2 2023-10-30T20:11:52.405 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T20:11:52.405 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:52.406 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:11:52.406 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:11:52.406 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:11:52.406 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:52.406 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:11:52.623 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-common-debuginfo 2023-10-30T20:11:53.168 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:02:20 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:11:53.310 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:11:53.311 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:53.312 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:11:53.312 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:53.312 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:11:53.313 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 410 M 2023-10-30T20:11:53.313 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:53.313 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:11:53.314 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:11:53.314 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:11:53.314 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:11:53.315 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 410 M 2023-10-30T20:11:53.315 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.7 G 2023-10-30T20:11:53.316 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:12:17.493 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-common-debuginfo-18.0.0-6839.gcc36dd85.el9 9.1 MB/s | 410 MB 00:44 2023-10-30T20:12:17.493 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:12:17.494 INFO:teuthology.orchestra.run.smithi183.stdout:Total 9.1 MB/s | 410 MB 00:44 2023-10-30T20:12:17.494 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:12:17.498 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:12:17.499 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:12:19.956 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:12:19.957 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:12:56.085 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-common-debuginfo-18.0.0-6839.gcc36dd85.el9 6.5 MB/s | 410 MB 01:02 2023-10-30T20:12:56.086 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:12:56.086 INFO:teuthology.orchestra.run.smithi069.stdout:Total 6.5 MB/s | 410 MB 01:02 2023-10-30T20:12:56.087 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:12:56.091 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:12:56.091 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:12:58.566 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:12:58.566 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:13:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:13:01.922 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:13:06.311 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:13:06.587 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:13:06.588 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:13:06.588 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:13:06.588 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:13:06.588 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:13:06.588 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:13:06.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-10-30T20:13:07.360 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:03:43 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:13:07.512 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:13:07.513 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2023-10-30T20:13:07.513 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:13:07.513 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2023-10-30T20:13:07.513 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:13:07.514 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.4 M 2023-10-30T20:13:07.514 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:13:07.514 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:13:07.514 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2023-10-30T20:13:07.515 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:13:07.515 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:13:07.515 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 1.4 M 2023-10-30T20:13:07.515 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 6.3 M 2023-10-30T20:13:07.515 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:13:09.025 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-68 974 kB/s | 1.4 MB 00:01 2023-10-30T20:13:09.026 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:13:09.026 INFO:teuthology.orchestra.run.smithi183.stdout:Total 973 kB/s | 1.4 MB 00:01 2023-10-30T20:13:09.026 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:13:09.030 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:13:09.030 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:13:09.074 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:13:09.075 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:13:09.329 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:13:09.422 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T20:13:09.846 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T20:13:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T20:13:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:13:10.121 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:13:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:13:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:13:10.121 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:13:10.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw-debuginfo 2023-10-30T20:13:10.828 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:03:46 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:13:10.969 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:13:10.970 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:13:10.970 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:13:10.970 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:13:10.970 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:13:10.970 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 174 M 2023-10-30T20:13:10.971 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:13:10.971 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:13:10.971 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:13:10.971 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:13:10.971 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:13:10.972 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 174 M 2023-10-30T20:13:10.972 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 732 M 2023-10-30T20:13:10.972 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:13:27.967 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-radosgw-debuginfo-18.0.0-6839.gcc36dd85.el 10 MB/s | 174 MB 00:16 2023-10-30T20:13:27.968 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:13:27.968 INFO:teuthology.orchestra.run.smithi183.stdout:Total 10 MB/s | 174 MB 00:16 2023-10-30T20:13:27.969 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:13:27.973 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:13:27.973 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:13:29.029 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:13:29.030 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:13:40.112 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:13:40.221 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:13:46.827 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:13:46.996 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T20:13:47.810 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:13:48.069 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:13:48.070 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:13:48.070 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:13:48.070 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:13:48.070 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:13:48.070 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:13:48.292 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-10-30T20:13:48.833 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:15 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:13:48.975 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:13:48.976 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2023-10-30T20:13:48.977 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:13:48.977 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2023-10-30T20:13:48.977 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:13:48.977 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.4 M 2023-10-30T20:13:48.977 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:13:48.978 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:13:48.978 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2023-10-30T20:13:48.978 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:13:48.978 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:13:48.978 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 1.4 M 2023-10-30T20:13:48.979 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 6.3 M 2023-10-30T20:13:48.979 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:13:49.292 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-68 4.6 MB/s | 1.4 MB 00:00 2023-10-30T20:13:49.293 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:13:49.293 INFO:teuthology.orchestra.run.smithi069.stdout:Total 4.5 MB/s | 1.4 MB 00:00 2023-10-30T20:13:49.293 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:13:49.296 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:13:49.296 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:13:49.337 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:13:49.337 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:13:49.591 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:13:49.698 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T20:13:50.092 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T20:13:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T20:13:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:13:50.351 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:13:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:13:50.352 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:13:50.352 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:13:50.539 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw-debuginfo 2023-10-30T20:13:51.078 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:18 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:13:51.218 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:13:51.219 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:13:51.219 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:13:51.220 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:13:51.220 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:13:51.220 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 174 M 2023-10-30T20:13:51.220 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:13:51.220 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:13:51.221 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:13:51.221 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:13:51.221 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:13:51.221 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 174 M 2023-10-30T20:13:51.221 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 732 M 2023-10-30T20:13:51.222 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:13:51.533 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T20:13:51.805 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T20:13:51.805 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:13:51.805 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:13:51.806 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:13:51.806 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:13:51.806 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:13:51.971 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-test-debuginfo 2023-10-30T20:13:52.521 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:28 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:13:52.663 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:13:52.664 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:13:52.664 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:13:52.664 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:13:52.665 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:13:52.665 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 626 M 2023-10-30T20:13:52.665 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:13:52.665 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:13:52.665 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:13:52.665 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:13:52.666 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:13:52.666 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 626 M 2023-10-30T20:13:52.666 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.8 G 2023-10-30T20:13:52.666 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:14:04.303 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-radosgw-debuginfo-18.0.0-6839.gcc36dd85.el 13 MB/s | 174 MB 00:13 2023-10-30T20:14:04.305 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:14:04.305 INFO:teuthology.orchestra.run.smithi069.stdout:Total 13 MB/s | 174 MB 00:13 2023-10-30T20:14:04.305 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:14:04.308 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:14:04.308 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:14:05.360 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:14:05.361 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:14:23.081 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:14:23.249 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T20:14:27.715 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T20:14:27.987 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T20:14:27.988 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:14:27.988 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:14:27.988 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:14:27.988 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:14:27.988 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:14:28.192 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-test-debuginfo 2023-10-30T20:14:28.735 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:55 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:14:28.878 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:14:28.879 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:14:28.879 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:14:28.879 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:14:28.879 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:14:28.879 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 626 M 2023-10-30T20:14:28.880 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:14:28.880 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:14:28.880 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:14:28.880 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:14:28.880 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:14:28.881 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 626 M 2023-10-30T20:14:28.881 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.8 G 2023-10-30T20:14:28.881 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:15:22.104 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-test-debuginfo-18.0.0-6839.gcc36dd85.el9.x 7.0 MB/s | 626 MB 01:29 2023-10-30T20:15:22.105 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:15:22.105 INFO:teuthology.orchestra.run.smithi183.stdout:Total 7.0 MB/s | 626 MB 01:29 2023-10-30T20:15:22.105 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:15:22.110 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:15:22.111 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:15:25.827 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:15:25.828 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:16:01.457 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-test-debuginfo-18.0.0-6839.gcc36dd85.el9.x 6.8 MB/s | 626 MB 01:32 2023-10-30T20:16:01.458 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:16:01.458 INFO:teuthology.orchestra.run.smithi069.stdout:Total 6.8 MB/s | 626 MB 01:32 2023-10-30T20:16:01.459 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:16:01.464 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:16:01.464 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:16:05.165 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:16:05.165 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:16:31.205 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:16:31.769 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:16:40.076 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:16:40.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:16:40.358 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:16:40.358 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:16:40.358 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:16:40.359 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:16:40.359 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:16:40.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-base-debuginfo 2023-10-30T20:16:41.178 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:07:17 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:16:41.271 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:16:41.322 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:16:41.322 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2023-10-30T20:16:41.322 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:16:41.386 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-debuginfo 2023-10-30T20:16:41.926 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:07:17 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:16:42.066 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:16:42.067 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:16:42.068 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:16:42.068 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:16:42.068 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:16:42.068 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 19 M 2023-10-30T20:16:42.068 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:16:42.069 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:16:42.069 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:16:42.069 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:16:42.069 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:16:42.069 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 19 M 2023-10-30T20:16:42.070 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 101 M 2023-10-30T20:16:42.070 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:16:47.529 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-mgr-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 3.5 MB/s | 19 MB 00:05 2023-10-30T20:16:47.530 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:16:47.531 INFO:teuthology.orchestra.run.smithi183.stdout:Total 3.5 MB/s | 19 MB 00:05 2023-10-30T20:16:47.531 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:16:47.535 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:16:47.535 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:16:47.682 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:16:47.683 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:16:49.857 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:16:49.960 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:16:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:16:51.181 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:16:51.181 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:16:51.182 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:16:51.182 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:16:51.182 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:16:51.182 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:16:51.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mds-debuginfo 2023-10-30T20:16:51.897 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:07:27 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:16:52.038 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:16:52.039 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:16:52.039 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:16:52.039 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:16:52.039 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:16:52.039 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 47 M 2023-10-30T20:16:52.040 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:16:52.040 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:16:52.040 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:16:52.040 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:16:52.040 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:16:52.041 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 47 M 2023-10-30T20:16:52.041 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 170 M 2023-10-30T20:16:52.041 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:17:10.589 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:17:10.756 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:17:11.279 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-mds-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 2.5 MB/s | 47 MB 00:19 2023-10-30T20:17:11.279 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:17:11.280 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.5 MB/s | 47 MB 00:19 2023-10-30T20:17:11.280 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:17:11.284 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:17:11.284 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:17:11.598 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:17:11.598 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:17:16.259 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:17:16.360 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:17:17.681 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:17:17.952 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:17:17.952 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:17:17.952 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:17:17.952 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:17:17.953 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:17:17.953 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:17:18.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mon-debuginfo 2023-10-30T20:17:18.673 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:07:54 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:17:18.815 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:17:18.816 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:17:18.817 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:17:18.817 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:17:18.817 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:17:18.817 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 112 M 2023-10-30T20:17:18.817 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:17:18.817 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:17:18.818 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:17:18.818 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:17:18.818 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:17:18.819 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 112 M 2023-10-30T20:17:18.819 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 509 M 2023-10-30T20:17:18.819 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:17:19.588 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:17:19.868 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:17:19.869 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:17:19.869 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:17:19.869 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:17:19.869 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:17:19.869 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:17:20.137 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-base-debuginfo 2023-10-30T20:17:20.682 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:07:47 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:17:20.774 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T20:17:20.823 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:17:20.824 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-10-30T20:17:20.824 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:17:20.904 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-debuginfo 2023-10-30T20:17:21.450 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:07:48 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:17:21.591 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:17:21.592 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:17:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:17:21.592 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:17:21.593 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:17:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 19 M 2023-10-30T20:17:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:17:21.593 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:17:21.593 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:17:21.593 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:17:21.594 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:17:21.594 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 19 M 2023-10-30T20:17:21.595 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 101 M 2023-10-30T20:17:21.595 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:17:24.317 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-mgr-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 7.1 MB/s | 19 MB 00:02 2023-10-30T20:17:24.318 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:17:24.319 INFO:teuthology.orchestra.run.smithi069.stdout:Total 7.1 MB/s | 19 MB 00:02 2023-10-30T20:17:24.319 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:17:24.323 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:17:24.323 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:17:24.469 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:17:24.470 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:17:26.650 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:17:26.765 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:17:27.722 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:17:27.992 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:17:27.992 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:17:27.993 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:17:27.993 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:17:27.993 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:17:27.993 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:17:28.183 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mds-debuginfo 2023-10-30T20:17:28.726 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:07:55 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:17:28.867 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:17:28.867 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:17:28.867 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:17:28.868 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:17:28.868 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:17:28.868 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 47 M 2023-10-30T20:17:28.868 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:17:28.868 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:17:28.869 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:17:28.869 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:17:28.869 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:17:28.869 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 47 M 2023-10-30T20:17:28.870 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 170 M 2023-10-30T20:17:28.870 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:17:37.161 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-mds-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 5.7 MB/s | 47 MB 00:08 2023-10-30T20:17:37.161 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:17:37.162 INFO:teuthology.orchestra.run.smithi069.stdout:Total 5.7 MB/s | 47 MB 00:08 2023-10-30T20:17:37.162 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:17:37.166 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:17:37.166 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:17:37.477 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:17:37.478 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:17:42.132 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:17:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:17:43.695 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:17:43.972 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:17:43.972 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:17:43.973 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:17:43.973 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:17:43.973 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:17:43.973 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:17:44.085 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-mon-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 4.4 MB/s | 112 MB 00:25 2023-10-30T20:17:44.086 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:17:44.087 INFO:teuthology.orchestra.run.smithi183.stdout:Total 4.4 MB/s | 112 MB 00:25 2023-10-30T20:17:44.087 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:17:44.091 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:17:44.091 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:17:44.151 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mon-debuginfo 2023-10-30T20:17:44.690 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:11 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:17:44.782 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:17:44.783 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:17:44.831 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:17:44.832 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:17:44.832 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:17:44.833 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:17:44.833 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:17:44.833 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 112 M 2023-10-30T20:17:44.833 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:17:44.833 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:17:44.834 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:17:44.834 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:17:44.834 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:17:44.834 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 112 M 2023-10-30T20:17:44.834 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 509 M 2023-10-30T20:17:44.834 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:17:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:17:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:17:59.699 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:17:59.984 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:17:59.984 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:17:59.984 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:17:59.984 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:17:59.985 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:17:59.985 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:18:00.140 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-osd-debuginfo 2023-10-30T20:18:00.683 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:08:36 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:18:00.823 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:18:00.823 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:18:00.823 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:18:00.824 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:18:00.824 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:18:00.824 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 315 M 2023-10-30T20:18:00.824 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:18:00.824 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:18:00.825 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:18:00.825 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:18:00.825 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:18:00.826 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 315 M 2023-10-30T20:18:00.826 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.4 G 2023-10-30T20:18:00.826 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:18:08.243 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-mon-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 4.8 MB/s | 112 MB 00:23 2023-10-30T20:18:08.244 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:18:08.244 INFO:teuthology.orchestra.run.smithi069.stdout:Total 4.8 MB/s | 112 MB 00:23 2023-10-30T20:18:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:18:08.248 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:18:08.249 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:18:08.939 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:18:08.939 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:18:20.657 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:18:20.824 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:18:24.169 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:18:24.459 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:18:24.459 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:18:24.460 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:18:24.460 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:18:24.460 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:18:24.460 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:18:24.617 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-osd-debuginfo 2023-10-30T20:18:25.159 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:52 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:18:25.298 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:18:25.299 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:18:25.299 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:18:25.300 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:18:25.300 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:18:25.300 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 315 M 2023-10-30T20:18:25.300 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:18:25.300 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:18:25.301 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:18:25.301 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:18:25.301 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:18:25.301 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 315 M 2023-10-30T20:18:25.301 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.4 G 2023-10-30T20:18:25.302 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:19:37.402 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-osd-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 4.4 MB/s | 315 MB 01:12 2023-10-30T20:19:37.403 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:19:37.403 INFO:teuthology.orchestra.run.smithi069.stdout:Total 4.4 MB/s | 315 MB 01:12 2023-10-30T20:19:37.404 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:19:37.407 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:19:37.407 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:19:39.296 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:19:39.296 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:19:50.503 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-osd-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 2.9 MB/s | 315 MB 01:49 2023-10-30T20:19:50.504 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:19:50.504 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.9 MB/s | 315 MB 01:49 2023-10-30T20:19:50.505 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:19:50.509 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:19:50.509 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:19:52.395 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:19:52.395 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:20:11.676 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:20:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:20:17.164 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:20:17.432 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:20:17.432 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:17.432 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:20:17.433 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:20:17.433 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:17.433 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:20:17.629 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-fuse-debuginfo 2023-10-30T20:20:18.169 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:10:45 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:20:18.309 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:20:18.310 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:18.310 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:20:18.310 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:18.310 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:20:18.311 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.8 M 2023-10-30T20:20:18.311 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:18.311 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:20:18.311 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:18.311 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:20:18.312 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:18.312 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 5.8 M 2023-10-30T20:20:18.312 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 28 M 2023-10-30T20:20:18.312 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:20:22.250 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-fuse-debuginfo-18.0.0-6839.gcc36dd85.el9.x 1.5 MB/s | 5.8 MB 00:03 2023-10-30T20:20:22.251 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:20:22.252 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.5 MB/s | 5.8 MB 00:03 2023-10-30T20:20:22.252 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:20:22.256 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:20:22.256 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:20:22.327 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:20:22.328 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:20:23.046 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:20:23.140 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:20:23.719 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:20:23.981 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:20:23.981 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:23.982 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:20:23.982 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:20:23.982 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:23.982 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:20:24.151 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados-devel-debuginfo 2023-10-30T20:20:24.674 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:20:24.696 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:10:51 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:20:24.830 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:20:24.836 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:20:24.837 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:24.837 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:20:24.837 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:24.838 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:20:24.838 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 108 k 2023-10-30T20:20:24.838 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:24.838 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:20:24.838 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:24.838 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:20:24.839 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:24.839 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 108 k 2023-10-30T20:20:24.839 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 440 k 2023-10-30T20:20:24.840 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:20:25.304 INFO:teuthology.orchestra.run.smithi069.stdout:librados-devel-debuginfo-18.0.0-6839.gcc36dd85. 233 kB/s | 108 kB 00:00 2023-10-30T20:20:25.304 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:20:25.305 INFO:teuthology.orchestra.run.smithi069.stdout:Total 233 kB/s | 108 kB 00:00 2023-10-30T20:20:25.305 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:20:25.307 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:20:25.308 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:20:25.341 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:20:25.341 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:20:25.457 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:20:25.552 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T20:20:25.930 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T20:20:26.197 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T20:20:26.197 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:26.197 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:20:26.198 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:20:26.198 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:26.198 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:20:26.375 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs2-debuginfo 2023-10-30T20:20:26.913 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:10:53 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:20:27.054 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:20:27.055 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:27.055 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:20:27.055 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:27.055 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:20:27.056 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.1 M 2023-10-30T20:20:27.056 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:27.056 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:20:27.056 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:27.056 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:20:27.056 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:27.057 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 5.1 M 2023-10-30T20:20:27.057 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 24 M 2023-10-30T20:20:27.057 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:20:28.281 INFO:teuthology.orchestra.run.smithi069.stdout:libcephfs2-debuginfo-18.0.0-6839.gcc36dd85.el9. 4.2 MB/s | 5.1 MB 00:01 2023-10-30T20:20:28.282 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:20:28.283 INFO:teuthology.orchestra.run.smithi069.stdout:Total 4.2 MB/s | 5.1 MB 00:01 2023-10-30T20:20:28.283 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:20:28.286 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:20:28.287 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:20:28.353 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:20:28.354 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:20:28.999 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:20:29.104 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T20:20:29.594 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T20:20:29.871 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T20:20:29.872 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:29.872 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:20:29.872 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:20:29.872 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:29.872 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:20:30.098 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados2-debuginfo 2023-10-30T20:20:30.642 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:10:57 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:20:30.784 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:20:30.785 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:30.785 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:20:30.785 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:30.785 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:20:30.786 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 45 M 2023-10-30T20:20:30.786 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:30.786 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:20:30.786 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:30.786 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:20:30.787 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:30.787 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 45 M 2023-10-30T20:20:30.787 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 255 M 2023-10-30T20:20:30.788 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:20:32.774 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:20:33.040 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:20:33.040 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:33.041 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:20:33.041 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:20:33.041 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:33.041 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:20:33.218 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-fuse-debuginfo 2023-10-30T20:20:33.768 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:11:09 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:20:33.918 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:20:33.918 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:20:33.919 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:20:33.919 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:20:33.919 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:20:33.919 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.8 M 2023-10-30T20:20:33.919 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:33.919 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:20:33.920 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:20:33.920 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:20:33.920 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:33.921 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 5.8 M 2023-10-30T20:20:33.921 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 28 M 2023-10-30T20:20:33.921 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:20:38.667 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-fuse-debuginfo-18.0.0-6839.gcc36dd85.el9.x 1.2 MB/s | 5.8 MB 00:04 2023-10-30T20:20:38.668 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:20:38.668 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.2 MB/s | 5.8 MB 00:04 2023-10-30T20:20:38.668 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:20:38.672 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:20:38.673 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:20:38.743 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:20:38.743 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:20:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:20:39.565 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:20:40.147 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:20:40.431 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:20:40.431 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:40.431 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:20:40.432 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:20:40.432 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:40.432 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:20:40.607 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados-devel-debuginfo 2023-10-30T20:20:41.151 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:11:17 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:20:41.291 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:20:41.292 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:20:41.292 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:20:41.292 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:20:41.293 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:20:41.293 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 108 k 2023-10-30T20:20:41.293 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:41.293 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:20:41.293 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:20:41.294 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:20:41.294 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:41.294 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 108 k 2023-10-30T20:20:41.294 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 440 k 2023-10-30T20:20:41.294 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:20:41.895 INFO:teuthology.orchestra.run.smithi183.stdout:librados-devel-debuginfo-18.0.0-6839.gcc36dd85. 180 kB/s | 108 kB 00:00 2023-10-30T20:20:41.895 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:20:41.896 INFO:teuthology.orchestra.run.smithi183.stdout:Total 180 kB/s | 108 kB 00:00 2023-10-30T20:20:41.896 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:20:41.899 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:20:41.899 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:20:41.933 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:20:41.934 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:20:42.047 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:20:42.151 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T20:20:42.536 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T20:20:42.815 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T20:20:42.815 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:42.815 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:20:42.815 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:20:42.816 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:42.816 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:20:42.970 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs2-debuginfo 2023-10-30T20:20:43.510 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:11:19 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:20:43.549 INFO:teuthology.orchestra.run.smithi069.stdout:librados2-debuginfo-18.0.0-6839.gcc36dd85.el9.x 3.5 MB/s | 45 MB 00:12 2023-10-30T20:20:43.550 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:20:43.551 INFO:teuthology.orchestra.run.smithi069.stdout:Total 3.5 MB/s | 45 MB 00:12 2023-10-30T20:20:43.551 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:20:43.555 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:20:43.555 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:20:43.650 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:20:43.650 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:20:43.651 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:20:43.651 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:20:43.651 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:20:43.651 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.1 M 2023-10-30T20:20:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:43.652 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:20:43.652 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:20:43.652 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:20:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:43.653 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 5.1 M 2023-10-30T20:20:43.653 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 24 M 2023-10-30T20:20:43.654 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:20:43.864 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:20:43.865 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:20:48.280 INFO:teuthology.orchestra.run.smithi183.stdout:libcephfs2-debuginfo-18.0.0-6839.gcc36dd85.el9. 1.1 MB/s | 5.1 MB 00:04 2023-10-30T20:20:48.281 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:20:48.281 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.1 MB/s | 5.1 MB 00:04 2023-10-30T20:20:48.282 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:20:48.285 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:20:48.286 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:20:48.352 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:20:48.352 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:20:49.005 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:20:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:20:49.097 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:20:49.111 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T20:20:49.694 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T20:20:49.969 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T20:20:49.970 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:49.970 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:20:49.970 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:20:49.970 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:49.971 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:20:50.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados2-debuginfo 2023-10-30T20:20:50.693 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:11:26 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:20:50.834 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:20:50.835 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:20:50.835 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:20:50.835 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:20:50.836 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:20:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 45 M 2023-10-30T20:20:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:50.836 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:20:50.836 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:20:50.836 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:20:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:20:50.837 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 45 M 2023-10-30T20:20:50.838 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 255 M 2023-10-30T20:20:50.838 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:20:50.959 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:20:51.235 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:20:51.235 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:51.235 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:20:51.235 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:20:51.235 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:51.236 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:20:51.448 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librbd1-debuginfo 2023-10-30T20:20:51.990 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:11:18 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:20:52.130 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:20:52.131 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:52.131 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:20:52.131 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:52.131 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:20:52.132 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 51 M 2023-10-30T20:20:52.132 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:52.132 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:20:52.132 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:20:52.132 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:20:52.133 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:20:52.133 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 51 M 2023-10-30T20:20:52.133 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 298 M 2023-10-30T20:20:52.134 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:21:00.715 INFO:teuthology.orchestra.run.smithi183.stdout:librados2-debuginfo-18.0.0-6839.gcc36dd85.el9.x 4.6 MB/s | 45 MB 00:09 2023-10-30T20:21:00.716 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:00.716 INFO:teuthology.orchestra.run.smithi183.stdout:Total 4.6 MB/s | 45 MB 00:09 2023-10-30T20:21:00.717 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:21:00.721 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:21:00.721 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:21:01.032 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:21:01.033 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:21:06.153 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:21:06.254 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:21:08.018 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:21:08.301 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T20:21:08.301 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:08.301 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:21:08.301 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:08.301 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:08.302 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:21:08.494 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librbd1-debuginfo 2023-10-30T20:21:09.041 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:11:45 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:21:09.181 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:21:09.182 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:09.182 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:21:09.182 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:09.182 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:21:09.183 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 51 M 2023-10-30T20:21:09.183 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:09.183 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:21:09.183 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:09.184 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:21:09.184 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:09.184 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 51 M 2023-10-30T20:21:09.184 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 298 M 2023-10-30T20:21:09.185 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:21:09.992 INFO:teuthology.orchestra.run.smithi069.stdout:librbd1-debuginfo-18.0.0-6839.gcc36dd85.el9.x86 2.8 MB/s | 51 MB 00:17 2023-10-30T20:21:09.993 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:09.993 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.8 MB/s | 51 MB 00:17 2023-10-30T20:21:09.993 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:21:09.997 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:21:09.997 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:21:10.336 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:21:10.336 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:21:14.842 INFO:teuthology.orchestra.run.smithi183.stdout:librbd1-debuginfo-18.0.0-6839.gcc36dd85.el9.x86 9.0 MB/s | 51 MB 00:05 2023-10-30T20:21:14.843 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:14.843 INFO:teuthology.orchestra.run.smithi183.stdout:Total 9.0 MB/s | 51 MB 00:05 2023-10-30T20:21:14.844 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:21:14.847 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:21:14.848 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:21:15.189 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:21:15.190 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:21:16.171 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:21:16.281 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:21:18.411 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:21:18.705 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:21:18.705 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:18.705 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:21:18.706 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:18.706 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:18.706 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:21:18.904 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-cephfs-debuginfo 2023-10-30T20:21:19.442 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:11:46 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:21:19.582 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:21:19.583 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:19.583 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:21:19.584 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:19.584 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:21:19.584 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 615 k 2023-10-30T20:21:19.584 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:19.584 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:21:19.585 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:19.585 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:21:19.585 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:19.585 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 615 k 2023-10-30T20:21:19.585 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.1 M 2023-10-30T20:21:19.585 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:21:20.193 INFO:teuthology.orchestra.run.smithi069.stdout:python3-cephfs-debuginfo-18.0.0-6839.gcc36dd85. 1.0 MB/s | 615 kB 00:00 2023-10-30T20:21:20.194 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:20.195 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.0 MB/s | 615 kB 00:00 2023-10-30T20:21:20.195 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:21:20.198 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:21:20.198 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:21:20.233 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:21:20.233 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:21:20.399 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:21:20.501 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T20:21:20.888 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T20:21:21.024 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:21:21.159 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T20:21:21.159 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:21.159 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:21:21.160 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:21.160 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:21.160 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:21:21.365 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rados-debuginfo 2023-10-30T20:21:21.390 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:21:21.904 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:11:48 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:21:22.043 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:21:22.044 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:22.044 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:21:22.045 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:22.045 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:21:22.045 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 770 k 2023-10-30T20:21:22.045 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:22.045 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:21:22.046 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:22.046 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:21:22.046 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:22.046 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 770 k 2023-10-30T20:21:22.046 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.7 M 2023-10-30T20:21:22.047 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:21:23.038 INFO:teuthology.orchestra.run.smithi069.stdout:python3-rados-debuginfo-18.0.0-6839.gcc36dd85.e 776 kB/s | 770 kB 00:00 2023-10-30T20:21:23.039 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:23.039 INFO:teuthology.orchestra.run.smithi069.stdout:Total 775 kB/s | 770 kB 00:00 2023-10-30T20:21:23.039 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:21:23.042 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:21:23.043 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:21:23.081 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:21:23.081 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:21:23.255 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:21:23.361 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T20:21:23.438 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:21:23.724 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:21:23.724 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:23.724 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:21:23.725 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:23.725 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:23.725 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:21:23.738 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T20:21:23.906 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-cephfs-debuginfo 2023-10-30T20:21:24.002 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T20:21:24.002 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:24.002 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:21:24.003 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:24.003 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:24.003 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:21:24.201 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rbd-debuginfo 2023-10-30T20:21:24.448 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:12:00 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:21:24.587 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:21:24.588 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:24.588 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:21:24.589 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:24.589 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:21:24.589 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 615 k 2023-10-30T20:21:24.589 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:24.589 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:21:24.590 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:24.590 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:21:24.590 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:24.590 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 615 k 2023-10-30T20:21:24.590 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.1 M 2023-10-30T20:21:24.591 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:21:24.740 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:11:51 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:21:24.869 INFO:teuthology.orchestra.run.smithi183.stdout:python3-cephfs-debuginfo-18.0.0-6839.gcc36dd85. 2.1 MB/s | 615 kB 00:00 2023-10-30T20:21:24.869 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:24.870 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.1 MB/s | 615 kB 00:00 2023-10-30T20:21:24.870 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:21:24.873 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:21:24.873 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:21:24.880 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:21:24.881 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:24.881 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:21:24.881 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:24.881 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:21:24.882 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.1 M 2023-10-30T20:21:24.882 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:24.882 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:21:24.882 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:24.882 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:21:24.883 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:24.883 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 1.1 M 2023-10-30T20:21:24.883 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 4.3 M 2023-10-30T20:21:24.885 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:21:24.909 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:21:24.909 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:21:25.071 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:21:25.175 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T20:21:25.516 INFO:teuthology.orchestra.run.smithi069.stdout:python3-rbd-debuginfo-18.0.0-6839.gcc36dd85.el9 1.8 MB/s | 1.1 MB 00:00 2023-10-30T20:21:25.517 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:25.517 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.8 MB/s | 1.1 MB 00:00 2023-10-30T20:21:25.517 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:21:25.520 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:21:25.520 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:21:25.560 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:21:25.560 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:21:25.567 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T20:21:25.776 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:21:25.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T20:21:25.846 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:25.846 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:21:25.847 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:25.847 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:25.847 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:21:25.881 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:21:26.019 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rados-debuginfo 2023-10-30T20:21:26.270 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:21:26.544 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:21:26.544 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:26.544 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:21:26.545 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:26.545 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:26.545 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:21:26.565 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:12:02 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:21:26.705 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:21:26.706 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:26.706 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:21:26.706 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:26.706 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:21:26.706 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 770 k 2023-10-30T20:21:26.707 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:26.707 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:21:26.707 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:26.707 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:21:26.707 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:26.708 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 770 k 2023-10-30T20:21:26.708 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.7 M 2023-10-30T20:21:26.708 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:21:26.755 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rgw-debuginfo 2023-10-30T20:21:27.297 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:11:54 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:21:27.436 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:21:27.437 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:27.437 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:21:27.438 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:27.438 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:21:27.438 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 337 k 2023-10-30T20:21:27.438 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:27.438 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:21:27.439 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:27.439 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:21:27.439 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:27.439 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 337 k 2023-10-30T20:21:27.439 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.0 M 2023-10-30T20:21:27.439 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:21:27.680 INFO:teuthology.orchestra.run.smithi183.stdout:python3-rados-debuginfo-18.0.0-6839.gcc36dd85.e 791 kB/s | 770 kB 00:00 2023-10-30T20:21:27.681 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:27.681 INFO:teuthology.orchestra.run.smithi183.stdout:Total 790 kB/s | 770 kB 00:00 2023-10-30T20:21:27.682 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:21:27.685 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:21:27.685 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:21:27.726 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:21:27.727 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:21:27.906 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:21:28.020 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T20:21:28.152 INFO:teuthology.orchestra.run.smithi069.stdout:python3-rgw-debuginfo-18.0.0-6839.gcc36dd85.el9 472 kB/s | 337 kB 00:00 2023-10-30T20:21:28.153 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:28.153 INFO:teuthology.orchestra.run.smithi069.stdout:Total 471 kB/s | 337 kB 00:00 2023-10-30T20:21:28.154 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:21:28.157 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:21:28.157 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:21:28.193 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:21:28.193 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:21:28.330 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:21:28.434 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T20:21:28.441 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:21:28.705 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T20:21:28.705 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:28.705 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:21:28.706 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:28.706 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:28.706 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:21:28.807 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:21:28.877 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rbd-debuginfo 2023-10-30T20:21:29.069 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:21:29.069 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:29.070 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:21:29.070 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:29.070 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:29.070 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:21:29.252 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-fuse-debuginfo 2023-10-30T20:21:29.415 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:12:05 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:21:29.555 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:21:29.556 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:29.556 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:21:29.556 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:29.557 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:21:29.557 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.1 M 2023-10-30T20:21:29.557 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:29.557 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:21:29.557 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:29.558 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:21:29.558 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:29.558 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 1.1 M 2023-10-30T20:21:29.558 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 4.3 M 2023-10-30T20:21:29.558 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:21:29.790 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:11:56 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:21:29.899 INFO:teuthology.orchestra.run.smithi183.stdout:python3-rbd-debuginfo-18.0.0-6839.gcc36dd85.el9 3.3 MB/s | 1.1 MB 00:00 2023-10-30T20:21:29.900 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:29.901 INFO:teuthology.orchestra.run.smithi183.stdout:Total 3.3 MB/s | 1.1 MB 00:00 2023-10-30T20:21:29.901 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:21:29.904 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:21:29.904 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:21:29.932 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:21:29.932 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:29.933 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:21:29.933 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:29.933 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:21:29.933 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 506 k 2023-10-30T20:21:29.933 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:29.934 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:21:29.934 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:29.934 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:21:29.934 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:29.935 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 506 k 2023-10-30T20:21:29.935 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.5 M 2023-10-30T20:21:29.935 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:21:29.944 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:21:29.944 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:21:30.175 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:21:30.222 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-fuse-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 1.7 MB/s | 506 kB 00:00 2023-10-30T20:21:30.222 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:30.222 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.7 MB/s | 506 kB 00:00 2023-10-30T20:21:30.223 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:21:30.226 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:21:30.226 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:21:30.263 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:21:30.264 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:21:30.273 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:21:30.418 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:21:30.520 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:21:30.693 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:21:30.937 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:21:30.981 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:21:30.981 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:30.981 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:21:30.981 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:30.982 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:30.982 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:21:31.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rgw-debuginfo 2023-10-30T20:21:31.194 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:21:31.194 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:31.194 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:21:31.194 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:31.194 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:31.195 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:21:31.381 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-mirror-debuginfo 2023-10-30T20:21:31.690 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:12:07 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:21:31.830 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:21:31.831 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:31.831 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:21:31.832 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:31.832 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:21:31.832 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 337 k 2023-10-30T20:21:31.832 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:31.832 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:21:31.832 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:31.833 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:21:31.833 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:31.833 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 337 k 2023-10-30T20:21:31.833 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.0 M 2023-10-30T20:21:31.833 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:21:31.923 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:11:58 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:21:32.065 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:21:32.065 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:32.066 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:21:32.066 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:32.066 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:21:32.066 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 51 M 2023-10-30T20:21:32.066 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:32.067 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:21:32.067 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:21:32.067 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:21:32.067 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:32.068 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 51 M 2023-10-30T20:21:32.068 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 289 M 2023-10-30T20:21:32.068 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:21:32.095 INFO:teuthology.orchestra.run.smithi183.stdout:python3-rgw-debuginfo-18.0.0-6839.gcc36dd85.el9 1.3 MB/s | 337 kB 00:00 2023-10-30T20:21:32.095 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:32.096 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.2 MB/s | 337 kB 00:00 2023-10-30T20:21:32.096 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:21:32.099 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:21:32.099 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:21:32.135 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:21:32.135 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:21:32.269 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:21:32.376 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:21:32.758 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:21:33.039 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T20:21:33.039 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:33.039 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:21:33.040 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:33.040 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:33.040 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:21:33.192 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-fuse-debuginfo 2023-10-30T20:21:33.735 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:12:09 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:21:33.876 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:21:33.877 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:33.877 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:21:33.877 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:33.878 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:21:33.878 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 506 k 2023-10-30T20:21:33.879 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:33.879 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:21:33.879 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:33.880 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:21:33.880 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:33.880 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 506 k 2023-10-30T20:21:33.880 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.5 M 2023-10-30T20:21:33.880 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:21:34.156 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-fuse-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 1.8 MB/s | 506 kB 00:00 2023-10-30T20:21:34.156 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:34.157 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.8 MB/s | 506 kB 00:00 2023-10-30T20:21:34.157 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:21:34.160 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:21:34.160 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:21:34.197 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:21:34.197 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:21:34.357 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:21:34.455 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:21:34.880 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:21:35.172 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T20:21:35.172 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:35.172 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:21:35.172 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:35.173 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:35.173 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:21:35.329 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-mirror-debuginfo 2023-10-30T20:21:35.869 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:12:11 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:21:36.009 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:21:36.010 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:36.010 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:21:36.010 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:36.010 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:21:36.011 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 51 M 2023-10-30T20:21:36.011 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:36.011 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:21:36.011 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:36.012 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:21:36.012 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:36.012 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 51 M 2023-10-30T20:21:36.012 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 289 M 2023-10-30T20:21:36.012 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:21:38.717 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-mirror-debuginfo-18.0.0-6839.gcc36dd85.el9. 19 MB/s | 51 MB 00:02 2023-10-30T20:21:38.718 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:38.719 INFO:teuthology.orchestra.run.smithi183.stdout:Total 19 MB/s | 51 MB 00:02 2023-10-30T20:21:38.719 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:21:38.723 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:21:38.723 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:21:39.057 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:21:39.058 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:21:44.842 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:21:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T20:21:46.914 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T20:21:47.185 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T20:21:47.186 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:47.186 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:21:47.186 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:47.186 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:47.186 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:21:47.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-nbd-debuginfo 2023-10-30T20:21:47.926 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:12:23 ago on Mon 30 Oct 2023 08:09:24 PM UTC. 2023-10-30T20:21:48.067 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-10-30T20:21:48.068 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:48.068 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2023-10-30T20:21:48.069 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:48.069 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-10-30T20:21:48.069 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.2 M 2023-10-30T20:21:48.069 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:48.070 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-10-30T20:21:48.070 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-10-30T20:21:48.070 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2023-10-30T20:21:48.070 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:48.071 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 1.2 M 2023-10-30T20:21:48.071 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 5.4 M 2023-10-30T20:21:48.071 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-10-30T20:21:49.011 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-nbd-debuginfo-18.0.0-6839.gcc36dd85.el9.x86 1.3 MB/s | 1.2 MB 00:00 2023-10-30T20:21:49.011 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:49.012 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.3 MB/s | 1.2 MB 00:00 2023-10-30T20:21:49.012 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-10-30T20:21:49.016 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-10-30T20:21:49.016 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-10-30T20:21:49.060 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-10-30T20:21:49.060 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-10-30T20:21:49.314 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-10-30T20:21:49.425 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:21:49.867 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:21:50.135 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:21:50.136 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:50.136 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-10-30T20:21:50.136 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:50.136 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:21:50.136 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-10-30T20:21:50.325 DEBUG:teuthology.parallel:result is None 2023-10-30T20:21:51.064 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-mirror-debuginfo-18.0.0-6839.gcc36dd85.el9. 2.7 MB/s | 51 MB 00:18 2023-10-30T20:21:51.065 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:21:51.066 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.7 MB/s | 51 MB 00:18 2023-10-30T20:21:51.066 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:21:51.070 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:21:51.070 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:21:51.405 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:21:51.405 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:21:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:21:57.270 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T20:21:59.236 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T20:21:59.506 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T20:21:59.507 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:59.507 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:21:59.507 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:21:59.507 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:21:59.507 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:21:59.685 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-nbd-debuginfo 2023-10-30T20:22:00.230 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:12:27 ago on Mon 30 Oct 2023 08:09:33 PM UTC. 2023-10-30T20:22:00.371 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-30T20:22:00.372 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:22:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-10-30T20:22:00.372 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:22:00.373 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-10-30T20:22:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.2 M 2023-10-30T20:22:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:00.373 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-10-30T20:22:00.373 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-10-30T20:22:00.374 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-10-30T20:22:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:00.374 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 1.2 M 2023-10-30T20:22:00.374 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 5.4 M 2023-10-30T20:22:00.374 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-10-30T20:22:00.691 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-nbd-debuginfo-18.0.0-6839.gcc36dd85.el9.x86 3.9 MB/s | 1.2 MB 00:00 2023-10-30T20:22:00.692 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-10-30T20:22:00.692 INFO:teuthology.orchestra.run.smithi069.stdout:Total 3.9 MB/s | 1.2 MB 00:00 2023-10-30T20:22:00.692 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-10-30T20:22:00.695 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-10-30T20:22:00.696 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-10-30T20:22:00.735 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-10-30T20:22:00.735 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-10-30T20:22:00.962 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-10-30T20:22:01.064 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:22:01.457 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:22:01.712 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T20:22:01.712 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:01.713 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-10-30T20:22:01.713 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T20:22:01.713 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:01.713 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-30T20:22:01.895 DEBUG:teuthology.parallel:result is None 2023-10-30T20:22:01.896 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T20:22:02.069 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-30T20:22:02.090 INFO:teuthology.orchestra.run.smithi069.stdout:18.0.0-6839.gcc36dd85.el9 2023-10-30T20:22:02.091 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6839.gcc36dd85.el9 2023-10-30T20:22:02.091 INFO:teuthology.task.install:The correct ceph version 18.0.0-6839.gcc36dd85 is installed. 2023-10-30T20:22:02.091 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T20:22:02.249 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-30T20:22:02.273 INFO:teuthology.orchestra.run.smithi183.stdout:18.0.0-6839.gcc36dd85.el9 2023-10-30T20:22:02.273 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6839.gcc36dd85.el9 2023-10-30T20:22:02.273 INFO:teuthology.task.install:The correct ceph version 18.0.0-6839.gcc36dd85 is installed. 2023-10-30T20:22:02.274 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-30T20:22:02.274 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:02.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T20:22:02.302 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:22:02.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T20:22:02.343 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-30T20:22:02.343 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:02.343 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T20:22:02.370 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T20:22:02.436 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:22:02.437 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T20:22:02.467 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T20:22:02.535 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-30T20:22:02.535 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:02.535 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T20:22:02.562 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T20:22:02.628 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:22:02.628 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T20:22:02.656 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T20:22:02.724 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-30T20:22:02.725 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:02.725 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T20:22:02.751 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T20:22:02.817 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:22:02.817 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T20:22:02.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T20:22:02.912 INFO:teuthology.run_tasks:Running task ceph... 2023-10-30T20:22:02.977 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-30T20:22:02.977 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /var/log/ceph 2023-10-30T20:22:02.980 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /var/log/ceph 2023-10-30T20:22:03.007 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-30T20:22:03.007 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T20:22:03.046 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T20:22:03.073 INFO:tasks.ceph:Creating extra log directories... 2023-10-30T20:22:03.073 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T20:22:03.112 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T20:22:03.149 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-30T20:22:03.149 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-30T20:22:03.150 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7441403', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': 'd0db9f2f95b9d6344950185071cace2a50d296eb'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 90, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 1876, 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d', 'sleep_before_teardown': 0, 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd0db9f2f95b9d6344950185071cace2a50d296eb', 'targets': {'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPYOadPJGZ6Dgmh4cUhlkNoyy6TxoaN3szWWY15zj2N5X3SpqTC5b49HSfaNZti1N2YwzFEGieaf0YsyG3qHro=', 'smithi183.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBDn5rofjGLsweyWrhB4yDSdrLYw67ggLPiGlfBteC8yQzlu+mrd5uMp5UD/byLTDpd2QtVo9ZCEH6z8FMVeIQk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-30_17:23:32', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-30T20:22:03.151 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T20:22:03.178 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T20:22:03.194 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T20:22:03.232 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T20:22:03.260 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:03.261 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T20:22:03.311 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T20:22:03.311 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2023-10-30T20:22:03.365 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T20:22:03.365 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T20:22:03.366 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-10-30T20:22:03.366 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T20:22:03.366 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T20:22:03.366 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-10-30 20:11:21.276435435 +0000 2023-10-30T20:22:03.367 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-10-30 20:08:34.202290697 +0000 2023-10-30T20:22:03.367 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-10-30 20:08:34.202290697 +0000 2023-10-30T20:22:03.367 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2023-10-30 20:08:34.202290697 +0000 2023-10-30T20:22:03.367 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T20:22:03.430 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-10-30T20:22:03.430 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-10-30T20:22:03.430 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000179881 s, 2.8 MB/s 2023-10-30T20:22:03.431 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T20:22:03.486 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2023-10-30T20:22:03.540 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T20:22:03.540 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T20:22:03.540 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-10-30T20:22:03.541 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T20:22:03.541 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T20:22:03.541 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-10-30 20:11:21.277435413 +0000 2023-10-30T20:22:03.541 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-10-30 20:08:34.709279017 +0000 2023-10-30T20:22:03.541 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-10-30 20:08:34.709279017 +0000 2023-10-30T20:22:03.542 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2023-10-30 20:08:34.709279017 +0000 2023-10-30T20:22:03.542 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T20:22:03.605 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-10-30T20:22:03.605 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-10-30T20:22:03.605 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.00018872 s, 2.7 MB/s 2023-10-30T20:22:03.606 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T20:22:03.662 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2023-10-30T20:22:03.716 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T20:22:03.716 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T20:22:03.716 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-10-30T20:22:03.717 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T20:22:03.717 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T20:22:03.717 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-10-30 20:11:21.277435413 +0000 2023-10-30T20:22:03.717 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-10-30 20:08:35.186268029 +0000 2023-10-30T20:22:03.717 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-10-30 20:08:35.186268029 +0000 2023-10-30T20:22:03.718 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2023-10-30 20:08:35.186268029 +0000 2023-10-30T20:22:03.718 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T20:22:03.781 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-10-30T20:22:03.781 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-10-30T20:22:03.781 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000198773 s, 2.6 MB/s 2023-10-30T20:22:03.782 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T20:22:03.838 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2023-10-30T20:22:03.892 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T20:22:03.892 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T20:22:03.892 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-10-30T20:22:03.892 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T20:22:03.893 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T20:22:03.893 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-10-30 20:11:21.278435390 +0000 2023-10-30T20:22:03.893 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-10-30 20:08:35.677256719 +0000 2023-10-30T20:22:03.893 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-10-30 20:08:35.677256719 +0000 2023-10-30T20:22:03.893 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2023-10-30 20:08:35.677256719 +0000 2023-10-30T20:22:03.894 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T20:22:03.956 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-10-30T20:22:03.957 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-10-30T20:22:03.957 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000183412 s, 2.8 MB/s 2023-10-30T20:22:03.958 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T20:22:04.013 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T20:22:04.013 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:22:04.013 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T20:22:04.029 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T20:22:04.030 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2023-10-30T20:22:04.085 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T20:22:04.085 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T20:22:04.086 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-10-30T20:22:04.086 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T20:22:04.086 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T20:22:04.086 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2023-10-30 20:11:07.970391044 +0000 2023-10-30T20:22:04.086 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2023-10-30 20:08:34.203588503 +0000 2023-10-30T20:22:04.087 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2023-10-30 20:08:34.203588503 +0000 2023-10-30T20:22:04.087 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2023-10-30 20:08:34.203588503 +0000 2023-10-30T20:22:04.087 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T20:22:04.151 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2023-10-30T20:22:04.152 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2023-10-30T20:22:04.152 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000208511 s, 2.5 MB/s 2023-10-30T20:22:04.153 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T20:22:04.209 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2023-10-30T20:22:04.264 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T20:22:04.264 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T20:22:04.264 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-10-30T20:22:04.265 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T20:22:04.265 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T20:22:04.265 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2023-10-30 20:11:07.970391044 +0000 2023-10-30T20:22:04.265 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2023-10-30 20:08:34.715575377 +0000 2023-10-30T20:22:04.265 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2023-10-30 20:08:34.715575377 +0000 2023-10-30T20:22:04.265 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2023-10-30 20:08:34.715575377 +0000 2023-10-30T20:22:04.266 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T20:22:04.329 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2023-10-30T20:22:04.330 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2023-10-30T20:22:04.330 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000209421 s, 2.4 MB/s 2023-10-30T20:22:04.331 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T20:22:04.386 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2023-10-30T20:22:04.442 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T20:22:04.442 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T20:22:04.442 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-10-30T20:22:04.442 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T20:22:04.443 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T20:22:04.443 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2023-10-30 20:11:07.970391044 +0000 2023-10-30T20:22:04.443 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2023-10-30 20:08:35.194563096 +0000 2023-10-30T20:22:04.443 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2023-10-30 20:08:35.194563096 +0000 2023-10-30T20:22:04.443 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2023-10-30 20:08:35.194563096 +0000 2023-10-30T20:22:04.444 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T20:22:04.507 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2023-10-30T20:22:04.507 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2023-10-30T20:22:04.508 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000264877 s, 1.9 MB/s 2023-10-30T20:22:04.509 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T20:22:04.564 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2023-10-30T20:22:04.618 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T20:22:04.618 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T20:22:04.618 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-10-30T20:22:04.619 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T20:22:04.619 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T20:22:04.619 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2023-10-30 20:11:07.971391018 +0000 2023-10-30T20:22:04.619 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2023-10-30 20:08:35.679550662 +0000 2023-10-30T20:22:04.619 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2023-10-30 20:08:35.679550662 +0000 2023-10-30T20:22:04.619 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2023-10-30 20:08:35.679550662 +0000 2023-10-30T20:22:04.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T20:22:04.683 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2023-10-30T20:22:04.683 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2023-10-30T20:22:04.683 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000223086 s, 2.3 MB/s 2023-10-30T20:22:04.684 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T20:22:04.739 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T20:22:04.739 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T20:22:04.739 INFO:tasks.ceph:Generating config... 2023-10-30T20:22:04.741 INFO:tasks.ceph:[client] debug monc = 20 2023-10-30T20:22:04.741 INFO:tasks.ceph:[client] debug ms = 1 2023-10-30T20:22:04.741 INFO:tasks.ceph:[client] debug objecter = 20 2023-10-30T20:22:04.741 INFO:tasks.ceph:[client] debug rados = 20 2023-10-30T20:22:04.741 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-30T20:22:04.742 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-10-30T20:22:04.742 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-10-30T20:22:04.742 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-10-30T20:22:04.742 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-30T20:22:04.743 INFO:tasks.ceph:[global] ms type = async 2023-10-30T20:22:04.743 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-10-30T20:22:04.743 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-30T20:22:04.743 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-30T20:22:04.744 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-30T20:22:04.744 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-30T20:22:04.744 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-30T20:22:04.744 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-10-30T20:22:04.744 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-10-30T20:22:04.745 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-10-30T20:22:04.745 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-10-30T20:22:04.745 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-10-30T20:22:04.745 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-10-30T20:22:04.746 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-10-30T20:22:04.746 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-10-30T20:22:04.746 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-30T20:22:04.746 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2023-10-30T20:22:04.747 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-30T20:22:04.747 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-30T20:22:04.747 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-10-30T20:22:04.747 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-10-30T20:22:04.748 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-10-30T20:22:04.748 INFO:tasks.ceph:[osd] debug monc = 20 2023-10-30T20:22:04.748 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-30T20:22:04.748 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-30T20:22:04.749 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-30T20:22:04.749 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-30T20:22:04.749 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-30T20:22:04.749 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-30T20:22:04.750 INFO:tasks.ceph:[osd] osd class default list = * 2023-10-30T20:22:04.750 INFO:tasks.ceph:[osd] osd class load list = * 2023-10-30T20:22:04.750 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-10-30T20:22:04.750 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-10-30T20:22:04.751 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-10-30T20:22:04.751 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-30T20:22:04.751 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-10-30T20:22:04.751 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-10-30T20:22:04.751 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-10-30T20:22:04.752 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-30T20:22:04.752 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-10-30T20:22:04.752 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-10-30T20:22:04.752 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-30T20:22:04.753 INFO:tasks.ceph:Setting up mon.a... 2023-10-30T20:22:04.753 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-30T20:22:04.803 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.keyring 2023-10-30T20:22:04.805 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-30T20:22:04.851 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T20:22:04.877 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.69'), ('mon.c', '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]'), ('mon.b', '172.21.15.183')] 2023-10-30T20:22:04.878 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790],172.21.15.183', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-10-30T20:22:04.879 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:04.879 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T20:22:04.932 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.69 --addv c '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]' --add b 172.21.15.183 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-30T20:22:05.015 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-30T20:22:05.015 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: generated fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:22:05.016 INFO:teuthology.orchestra.run.smithi069.stdout:setting min_mon_release = pacific 2023-10-30T20:22:05.016 INFO:teuthology.orchestra.run.smithi069.stdout:epoch 0 2023-10-30T20:22:05.016 INFO:teuthology.orchestra.run.smithi069.stdout:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:22:05.016 INFO:teuthology.orchestra.run.smithi069.stdout:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:22:05.016 INFO:teuthology.orchestra.run.smithi069.stdout:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:22:05.017 INFO:teuthology.orchestra.run.smithi069.stdout:min_mon_release 16 (pacific) 2023-10-30T20:22:05.017 INFO:teuthology.orchestra.run.smithi069.stdout:election_strategy: 3 2023-10-30T20:22:05.017 INFO:teuthology.orchestra.run.smithi069.stdout:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:22:05.017 INFO:teuthology.orchestra.run.smithi069.stdout:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:22:05.017 INFO:teuthology.orchestra.run.smithi069.stdout:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:22:05.017 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-30T20:22:05.018 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T20:22:05.072 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 32c1a723-7694-47d8-8b37-b1257dfe1796... 2023-10-30T20:22:05.073 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T20:22:05.114 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T20:22:05.161 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2023-10-30T20:22:05.161 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2023-10-30T20:22:05.162 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T20:22:05.162 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2023-10-30T20:22:05.162 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.162 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2023-10-30T20:22:05.162 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.163 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2023-10-30T20:22:05.163 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.163 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2023-10-30T20:22:05.163 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2023-10-30T20:22:05.163 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.164 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2023-10-30T20:22:05.164 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2023-10-30T20:22:05.164 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.164 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T20:22:05.164 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2023-10-30T20:22:05.165 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.165 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T20:22:05.165 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T20:22:05.165 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.165 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2023-10-30T20:22:05.166 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.166 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2023-10-30T20:22:05.166 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2023-10-30T20:22:05.166 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2023-10-30T20:22:05.166 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2023-10-30T20:22:05.167 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2023-10-30T20:22:05.167 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2023-10-30T20:22:05.167 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2023-10-30T20:22:05.167 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T20:22:05.167 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T20:22:05.167 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2023-10-30T20:22:05.168 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.168 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T20:22:05.168 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.168 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2023-10-30T20:22:05.168 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.169 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2023-10-30T20:22:05.169 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.169 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2023-10-30T20:22:05.169 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2023-10-30T20:22:05.169 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2023-10-30T20:22:05.169 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790],172.21.15.183" 2023-10-30T20:22:05.170 INFO:teuthology.orchestra.run.smithi069.stdout: mon client directed command retry = 5 2023-10-30T20:22:05.170 INFO:teuthology.orchestra.run.smithi069.stdout: mon election default strategy = 3 2023-10-30T20:22:05.170 INFO:teuthology.orchestra.run.smithi069.stdout: ms bind msgr1 = False 2023-10-30T20:22:05.170 INFO:teuthology.orchestra.run.smithi069.stdout: ms bind msgr2 = True 2023-10-30T20:22:05.170 INFO:teuthology.orchestra.run.smithi069.stdout: ms inject socket failures = 5000 2023-10-30T20:22:05.171 INFO:teuthology.orchestra.run.smithi069.stdout: ms type = async 2023-10-30T20:22:05.171 INFO:teuthology.orchestra.run.smithi069.stdout: osd heartbeat grace = 80 2023-10-30T20:22:05.171 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:22:05.171 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.171 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2023-10-30T20:22:05.172 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2023-10-30T20:22:05.172 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.172 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2023-10-30T20:22:05.172 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2023-10-30T20:22:05.172 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2023-10-30T20:22:05.172 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.173 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2023-10-30T20:22:05.173 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2023-10-30T20:22:05.173 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.173 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2023-10-30T20:22:05.173 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2023-10-30T20:22:05.174 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2023-10-30T20:22:05.174 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.174 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2023-10-30T20:22:05.174 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2023-10-30T20:22:05.174 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.174 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2023-10-30T20:22:05.175 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.175 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2023-10-30T20:22:05.175 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.175 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2023-10-30T20:22:05.175 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.176 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2023-10-30T20:22:05.176 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2023-10-30T20:22:05.176 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.176 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2023-10-30T20:22:05.176 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2023-10-30T20:22:05.176 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2023-10-30T20:22:05.177 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore compression algorithm = lz4 2023-10-30T20:22:05.177 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore compression mode = aggressive 2023-10-30T20:22:05.177 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2023-10-30T20:22:05.177 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore zero block detection = True 2023-10-30T20:22:05.177 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1 2023-10-30T20:22:05.178 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1 2023-10-30T20:22:05.178 INFO:teuthology.orchestra.run.smithi069.stdout: debug monc = 20 2023-10-30T20:22:05.178 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-10-30T20:22:05.178 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2023-10-30T20:22:05.178 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 10 2023-10-30T20:22:05.179 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T20:22:05.179 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2023-10-30T20:22:05.179 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2023-10-30T20:22:05.179 INFO:teuthology.orchestra.run.smithi069.stdout: osd class default list = * 2023-10-30T20:22:05.179 INFO:teuthology.orchestra.run.smithi069.stdout: osd class load list = * 2023-10-30T20:22:05.179 INFO:teuthology.orchestra.run.smithi069.stdout: osd client watch timeout = 120 2023-10-30T20:22:05.180 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify cached snaps = True 2023-10-30T20:22:05.180 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify missing on start = True 2023-10-30T20:22:05.180 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2023-10-30T20:22:05.180 INFO:teuthology.orchestra.run.smithi069.stdout: osd fast shutdown = False 2023-10-30T20:22:05.180 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock override recovery settings = True 2023-10-30T20:22:05.181 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2023-10-30T20:22:05.181 INFO:teuthology.orchestra.run.smithi069.stdout: osd op queue = debug_random 2023-10-30T20:22:05.181 INFO:teuthology.orchestra.run.smithi069.stdout: osd op queue cut off = debug_random 2023-10-30T20:22:05.181 INFO:teuthology.orchestra.run.smithi069.stdout: osd shutdown pgref assert = True 2023-10-30T20:22:05.181 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.181 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2023-10-30T20:22:05.182 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-10-30T20:22:05.182 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2023-10-30T20:22:05.182 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2023-10-30T20:22:05.182 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2023-10-30T20:22:05.182 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2023-10-30T20:22:05.183 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2023-10-30T20:22:05.183 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2023-10-30T20:22:05.183 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.183 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2023-10-30T20:22:05.183 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-10-30T20:22:05.184 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2023-10-30T20:22:05.184 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2023-10-30T20:22:05.184 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2023-10-30T20:22:05.184 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2023-10-30T20:22:05.184 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2023-10-30T20:22:05.184 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2023-10-30T20:22:05.185 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2023-10-30T20:22:05.185 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2023-10-30T20:22:05.185 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2023-10-30T20:22:05.185 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.185 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T20:22:05.186 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T20:22:05.186 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T20:22:05.186 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.186 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T20:22:05.186 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T20:22:05.186 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T20:22:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2023-10-30T20:22:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2023-10-30T20:22:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2023-10-30T20:22:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: mon min osdmap epochs = 50 2023-10-30T20:22:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd crush smoke test = False 2023-10-30T20:22:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: mon osdmap full prune interval = 2 2023-10-30T20:22:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: mon osdmap full prune min = 15 2023-10-30T20:22:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: mon osdmap full prune txsize = 2 2023-10-30T20:22:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: mon scrub interval = 300 2023-10-30T20:22:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on pool no app = False 2023-10-30T20:22:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: paxos service trim min = 10 2023-10-30T20:22:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:22:05.189 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2023-10-30T20:22:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2023-10-30T20:22:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2023-10-30T20:22:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2023-10-30T20:22:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T20:22:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T20:22:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: debug monc = 20 2023-10-30T20:22:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-10-30T20:22:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: debug objecter = 20 2023-10-30T20:22:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: debug rados = 20 2023-10-30T20:22:05.191 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2023-10-30T20:22:05.191 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2023-10-30T20:22:05.191 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2023-10-30T20:22:05.193 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2023-10-30T20:22:05.193 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2023-10-30T20:22:05.193 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T20:22:05.194 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2023-10-30T20:22:05.194 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.194 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2023-10-30T20:22:05.194 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.194 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2023-10-30T20:22:05.194 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.195 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2023-10-30T20:22:05.195 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2023-10-30T20:22:05.195 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.195 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2023-10-30T20:22:05.195 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2023-10-30T20:22:05.196 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.196 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T20:22:05.196 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2023-10-30T20:22:05.196 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.196 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T20:22:05.197 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T20:22:05.197 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.197 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2023-10-30T20:22:05.197 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.197 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2023-10-30T20:22:05.197 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2023-10-30T20:22:05.198 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2023-10-30T20:22:05.198 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2023-10-30T20:22:05.198 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2023-10-30T20:22:05.198 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2023-10-30T20:22:05.198 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2023-10-30T20:22:05.199 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T20:22:05.199 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T20:22:05.199 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2023-10-30T20:22:05.199 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.199 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T20:22:05.200 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.200 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2023-10-30T20:22:05.200 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.200 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2023-10-30T20:22:05.200 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.201 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2023-10-30T20:22:05.201 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2023-10-30T20:22:05.201 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2023-10-30T20:22:05.201 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = "172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790],172.21.15.183" 2023-10-30T20:22:05.201 INFO:teuthology.orchestra.run.smithi183.stdout: mon client directed command retry = 5 2023-10-30T20:22:05.201 INFO:teuthology.orchestra.run.smithi183.stdout: mon election default strategy = 3 2023-10-30T20:22:05.202 INFO:teuthology.orchestra.run.smithi183.stdout: ms bind msgr1 = False 2023-10-30T20:22:05.202 INFO:teuthology.orchestra.run.smithi183.stdout: ms bind msgr2 = True 2023-10-30T20:22:05.202 INFO:teuthology.orchestra.run.smithi183.stdout: ms inject socket failures = 5000 2023-10-30T20:22:05.202 INFO:teuthology.orchestra.run.smithi183.stdout: ms type = async 2023-10-30T20:22:05.202 INFO:teuthology.orchestra.run.smithi183.stdout: osd heartbeat grace = 80 2023-10-30T20:22:05.203 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:22:05.203 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.203 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2023-10-30T20:22:05.203 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2023-10-30T20:22:05.203 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.203 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2023-10-30T20:22:05.204 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2023-10-30T20:22:05.204 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2023-10-30T20:22:05.204 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.204 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2023-10-30T20:22:05.204 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2023-10-30T20:22:05.205 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.205 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2023-10-30T20:22:05.205 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2023-10-30T20:22:05.205 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2023-10-30T20:22:05.205 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.205 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2023-10-30T20:22:05.206 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2023-10-30T20:22:05.206 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.206 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2023-10-30T20:22:05.206 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.206 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2023-10-30T20:22:05.207 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.207 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2023-10-30T20:22:05.207 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.207 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2023-10-30T20:22:05.207 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2023-10-30T20:22:05.207 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.208 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2023-10-30T20:22:05.208 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2023-10-30T20:22:05.208 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore block size = 96636764160 2023-10-30T20:22:05.208 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore compression algorithm = lz4 2023-10-30T20:22:05.208 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore compression mode = aggressive 2023-10-30T20:22:05.209 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore fsck on mount = True 2023-10-30T20:22:05.209 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore zero block detection = True 2023-10-30T20:22:05.209 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 1 2023-10-30T20:22:05.209 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 1 2023-10-30T20:22:05.209 INFO:teuthology.orchestra.run.smithi183.stdout: debug monc = 20 2023-10-30T20:22:05.209 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2023-10-30T20:22:05.210 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 20 2023-10-30T20:22:05.210 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 10 2023-10-30T20:22:05.210 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T20:22:05.210 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd full ratio = 0.9 2023-10-30T20:22:05.210 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd nearfull ratio = 0.8 2023-10-30T20:22:05.211 INFO:teuthology.orchestra.run.smithi183.stdout: osd class default list = * 2023-10-30T20:22:05.211 INFO:teuthology.orchestra.run.smithi183.stdout: osd class load list = * 2023-10-30T20:22:05.211 INFO:teuthology.orchestra.run.smithi183.stdout: osd client watch timeout = 120 2023-10-30T20:22:05.211 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify cached snaps = True 2023-10-30T20:22:05.211 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify missing on start = True 2023-10-30T20:22:05.211 INFO:teuthology.orchestra.run.smithi183.stdout: osd failsafe full ratio = 0.95 2023-10-30T20:22:05.212 INFO:teuthology.orchestra.run.smithi183.stdout: osd fast shutdown = False 2023-10-30T20:22:05.212 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock override recovery settings = True 2023-10-30T20:22:05.212 INFO:teuthology.orchestra.run.smithi183.stdout: osd objectstore = bluestore 2023-10-30T20:22:05.212 INFO:teuthology.orchestra.run.smithi183.stdout: osd op queue = debug_random 2023-10-30T20:22:05.212 INFO:teuthology.orchestra.run.smithi183.stdout: osd op queue cut off = debug_random 2023-10-30T20:22:05.213 INFO:teuthology.orchestra.run.smithi183.stdout: osd shutdown pgref assert = True 2023-10-30T20:22:05.213 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.213 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2023-10-30T20:22:05.213 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2023-10-30T20:22:05.213 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2023-10-30T20:22:05.214 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2023-10-30T20:22:05.214 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2023-10-30T20:22:05.214 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2023-10-30T20:22:05.214 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2023-10-30T20:22:05.214 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2023-10-30T20:22:05.215 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.215 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2023-10-30T20:22:05.215 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2023-10-30T20:22:05.215 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2023-10-30T20:22:05.215 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2023-10-30T20:22:05.215 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2023-10-30T20:22:05.216 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2023-10-30T20:22:05.216 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2023-10-30T20:22:05.216 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2023-10-30T20:22:05.216 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2023-10-30T20:22:05.216 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2023-10-30T20:22:05.217 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2023-10-30T20:22:05.217 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.217 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T20:22:05.217 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T20:22:05.217 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T20:22:05.217 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.218 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T20:22:05.218 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T20:22:05.218 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T20:22:05.218 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.218 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2023-10-30T20:22:05.219 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2023-10-30T20:22:05.219 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.219 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2023-10-30T20:22:05.219 INFO:teuthology.orchestra.run.smithi183.stdout: mon min osdmap epochs = 50 2023-10-30T20:22:05.219 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd crush smoke test = False 2023-10-30T20:22:05.219 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune interval = 2 2023-10-30T20:22:05.220 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune min = 15 2023-10-30T20:22:05.220 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune txsize = 2 2023-10-30T20:22:05.220 INFO:teuthology.orchestra.run.smithi183.stdout: mon scrub interval = 300 2023-10-30T20:22:05.220 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on pool no app = False 2023-10-30T20:22:05.220 INFO:teuthology.orchestra.run.smithi183.stdout: paxos service trim min = 10 2023-10-30T20:22:05.221 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-10-30T20:22:05.221 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2023-10-30T20:22:05.221 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2023-10-30T20:22:05.221 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2023-10-30T20:22:05.221 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2023-10-30T20:22:05.221 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T20:22:05.222 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T20:22:05.222 INFO:teuthology.orchestra.run.smithi183.stdout: debug monc = 20 2023-10-30T20:22:05.222 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2023-10-30T20:22:05.222 INFO:teuthology.orchestra.run.smithi183.stdout: debug objecter = 20 2023-10-30T20:22:05.222 INFO:teuthology.orchestra.run.smithi183.stdout: debug rados = 20 2023-10-30T20:22:05.223 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2023-10-30T20:22:05.223 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.c] 2023-10-30T20:22:05.223 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.b] 2023-10-30T20:22:05.223 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-30T20:22:05.224 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-30T20:22:05.288 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-30T20:22:05.288 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:05.288 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-30T20:22:05.303 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:05.303 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-30T20:22:05.357 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2023-10-30T20:22:05.357 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:05.358 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T20:22:05.358 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T20:22:05.436 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:05.436 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T20:22:05.490 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi183.front.sepia.ceph.com 2023-10-30T20:22:05.491 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:22:05.491 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T20:22:05.491 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T20:22:05.531 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:22:05.531 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T20:22:05.585 INFO:tasks.ceph:Setting up mon nodes... 2023-10-30T20:22:05.586 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-30T20:22:05.586 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T20:22:05.640 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T20:22:05.643 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T20:22:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T20:22:05.708 INFO:tasks.ceph:Setting up mds nodes... 2023-10-30T20:22:05.708 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-30T20:22:05.709 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-30T20:22:05.750 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-30T20:22:05.765 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-30T20:22:05.807 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-30T20:22:05.822 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-30T20:22:05.823 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T20:22:05.823 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-30T20:22:05.849 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T20:22:05.850 INFO:tasks.ceph:role: osd.0 2023-10-30T20:22:05.850 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com 2023-10-30T20:22:05.850 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T20:22:05.923 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T20:22:05.923 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T20:22:05.924 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T20:22:05.924 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T20:22:05.924 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T20:22:05.924 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-10-30T20:22:05.924 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T20:22:05.925 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T20:22:05.925 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T20:22:05.925 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T20:22:06.435 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2023-10-30T20:22:06.437 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-10-30T20:22:06.438 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-30T20:22:06.592 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-30T20:22:06.625 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-30T20:22:06.692 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T20:22:06.692 INFO:tasks.ceph:role: osd.1 2023-10-30T20:22:06.692 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com 2023-10-30T20:22:06.692 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T20:22:06.766 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T20:22:06.766 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T20:22:06.766 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T20:22:06.767 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T20:22:06.767 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T20:22:06.767 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-10-30T20:22:06.767 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T20:22:06.767 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T20:22:06.768 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T20:22:06.768 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T20:22:07.211 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2023-10-30T20:22:07.212 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-10-30T20:22:07.213 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-30T20:22:07.307 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-30T20:22:07.338 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-30T20:22:07.406 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T20:22:07.406 INFO:tasks.ceph:role: osd.2 2023-10-30T20:22:07.407 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com 2023-10-30T20:22:07.407 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T20:22:07.480 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T20:22:07.480 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T20:22:07.480 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T20:22:07.480 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T20:22:07.480 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T20:22:07.481 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-10-30T20:22:07.481 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T20:22:07.481 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T20:22:07.481 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T20:22:07.481 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T20:22:07.966 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2023-10-30T20:22:07.968 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-10-30T20:22:07.969 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-30T20:22:08.118 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-30T20:22:08.152 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-30T20:22:08.219 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T20:22:08.219 INFO:tasks.ceph:role: osd.3 2023-10-30T20:22:08.220 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com 2023-10-30T20:22:08.220 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T20:22:08.293 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T20:22:08.293 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T20:22:08.293 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T20:22:08.294 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T20:22:08.294 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T20:22:08.294 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-10-30T20:22:08.294 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T20:22:08.294 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T20:22:08.295 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T20:22:08.295 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T20:22:08.807 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2023-10-30T20:22:08.809 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-10-30T20:22:08.810 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-30T20:22:08.947 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-30T20:22:08.979 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T20:22:09.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:09.071+0000 7fbf25eb6500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-30T20:22:09.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:09.071+0000 7fbf25eb6500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-30T20:22:09.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:09.071+0000 7fbf25eb6500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T20:22:09.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:09.071+0000 7fbf25eb6500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T20:22:09.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:09.071+0000 7fbf25eb6500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T20:22:09.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:09.072+0000 7fbf25eb6500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-30T20:22:11.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-30T20:22:11.948 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T20:22:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:12.038+0000 7fd89381f500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-30T20:22:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:12.038+0000 7fd89381f500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-30T20:22:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:12.038+0000 7fd89381f500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T20:22:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:12.038+0000 7fd89381f500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T20:22:12.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:12.038+0000 7fd89381f500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T20:22:12.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:12.039+0000 7fd89381f500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-30T20:22:14.887 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-30T20:22:14.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T20:22:15.006 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:15.004+0000 7f90d7a23500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-30T20:22:15.006 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:15.004+0000 7f90d7a23500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-30T20:22:15.006 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:15.004+0000 7f90d7a23500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T20:22:15.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:15.004+0000 7f90d7a23500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T20:22:15.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:15.004+0000 7f90d7a23500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T20:22:15.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:15.005+0000 7f90d7a23500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-30T20:22:17.849 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-30T20:22:17.880 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T20:22:17.972 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:17.971+0000 7feb01b1c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-30T20:22:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:17.971+0000 7feb01b1c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-30T20:22:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:17.971+0000 7feb01b1c500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T20:22:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:17.971+0000 7feb01b1c500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T20:22:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:17.971+0000 7feb01b1c500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T20:22:17.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:17.972+0000 7feb01b1c500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-30T20:22:20.818 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-30T20:22:20.848 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-30T20:22:20.880 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T20:22:20.880 INFO:tasks.ceph:role: osd.4 2023-10-30T20:22:20.880 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com 2023-10-30T20:22:20.881 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T20:22:20.958 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T20:22:20.958 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T20:22:20.959 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T20:22:20.959 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T20:22:20.959 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T20:22:20.959 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2023-10-30T20:22:20.959 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T20:22:20.960 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T20:22:20.960 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T20:22:20.960 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T20:22:21.440 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2023-10-30T20:22:21.442 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2023-10-30T20:22:21.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-30T20:22:21.563 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-30T20:22:21.601 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-30T20:22:21.670 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T20:22:21.670 INFO:tasks.ceph:role: osd.5 2023-10-30T20:22:21.670 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com 2023-10-30T20:22:21.670 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T20:22:21.745 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T20:22:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T20:22:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T20:22:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T20:22:21.746 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T20:22:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2023-10-30T20:22:21.746 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T20:22:21.747 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T20:22:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T20:22:21.747 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T20:22:22.212 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2023-10-30T20:22:22.214 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2023-10-30T20:22:22.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-30T20:22:22.305 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-30T20:22:22.338 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-30T20:22:22.407 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T20:22:22.407 INFO:tasks.ceph:role: osd.6 2023-10-30T20:22:22.407 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com 2023-10-30T20:22:22.408 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T20:22:22.482 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T20:22:22.482 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T20:22:22.483 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T20:22:22.483 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T20:22:22.483 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T20:22:22.483 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2023-10-30T20:22:22.483 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T20:22:22.484 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T20:22:22.484 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T20:22:22.484 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T20:22:22.971 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2023-10-30T20:22:22.973 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2023-10-30T20:22:22.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-30T20:22:23.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-30T20:22:23.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-30T20:22:23.227 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T20:22:23.227 INFO:tasks.ceph:role: osd.7 2023-10-30T20:22:23.227 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi183.front.sepia.ceph.com 2023-10-30T20:22:23.228 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T20:22:23.303 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T20:22:23.303 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T20:22:23.304 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T20:22:23.304 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T20:22:23.304 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T20:22:23.304 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2023-10-30T20:22:23.304 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T20:22:23.305 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T20:22:23.305 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T20:22:23.305 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T20:22:23.814 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2023-10-30T20:22:23.816 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2023-10-30T20:22:23.816 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-30T20:22:23.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-30T20:22:23.972 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T20:22:24.074 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:24.072+0000 7f66a5c10500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-30T20:22:24.075 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:24.072+0000 7f66a5c10500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-30T20:22:24.075 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:24.072+0000 7f66a5c10500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T20:22:24.075 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:24.072+0000 7f66a5c10500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T20:22:24.076 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:24.072+0000 7f66a5c10500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T20:22:24.076 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:24.073+0000 7f66a5c10500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-30T20:22:26.922 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-30T20:22:26.953 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T20:22:27.045 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:27.044+0000 7fbe39c2b500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-30T20:22:27.045 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:27.044+0000 7fbe39c2b500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-30T20:22:27.046 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:27.044+0000 7fbe39c2b500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T20:22:27.046 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:27.044+0000 7fbe39c2b500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T20:22:27.046 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:27.044+0000 7fbe39c2b500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T20:22:27.046 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:27.045+0000 7fbe39c2b500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-30T20:22:29.884 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-30T20:22:29.915 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T20:22:30.007 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:30.005+0000 7fd060463500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-30T20:22:30.007 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:30.005+0000 7fd060463500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-30T20:22:30.007 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:30.005+0000 7fd060463500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T20:22:30.007 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:30.005+0000 7fd060463500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T20:22:30.007 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:30.005+0000 7fd060463500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T20:22:30.008 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:30.006+0000 7fd060463500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-30T20:22:32.851 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-30T20:22:32.882 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T20:22:32.974 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:32.973+0000 7f79967ad500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-30T20:22:32.975 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:32.973+0000 7f79967ad500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-30T20:22:32.975 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:32.973+0000 7f79967ad500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T20:22:32.975 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:32.973+0000 7f79967ad500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T20:22:32.975 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:32.973+0000 7f79967ad500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T20:22:32.975 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:22:32.974+0000 7f79967ad500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-30T20:22:35.813 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-30T20:22:35.844 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-30T20:22:35.844 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:35.844 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-30T20:22:35.876 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:35.876 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-30T20:22:35.946 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:35.947 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-30T20:22:36.017 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:36.017 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-30T20:22:36.087 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:36.087 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-30T20:22:36.156 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:22:36.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-30T20:22:36.188 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:22:36.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-30T20:22:36.258 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:22:36.259 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-30T20:22:36.329 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:22:36.329 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-30T20:22:36.398 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:22:36.398 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-30T20:22:36.466 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:22:36.466 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-30T20:22:36.484 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:22:36.484 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-30T20:22:36.522 INFO:tasks.ceph:Adding keys to all mons... 2023-10-30T20:22:36.522 DEBUG:teuthology.orchestra.run.smithi069:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T20:22:36.526 DEBUG:teuthology.orchestra.run.smithi183:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T20:22:36.565 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.y] 2023-10-30T20:22:36.566 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBtEEBls4YpJhAAlhztShxQj/FfHL1kwBA8zQ== 2023-10-30T20:22:36.566 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2023-10-30T20:22:36.566 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBxEEBlhftNBBAA9B+qy/0F2oAh1LIejBSnIQ== 2023-10-30T20:22:36.566 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2023-10-30T20:22:36.566 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQB0EEBlRxVbAhAAvDKnQvBYMEK89xrxDZbFiA== 2023-10-30T20:22:36.567 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2023-10-30T20:22:36.567 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQB3EEBl3mpUABAAfn4Y9gQwt4Ab+O2c0A2McQ== 2023-10-30T20:22:36.567 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.3] 2023-10-30T20:22:36.567 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQB5EEBlBx/yORAAUWI0GBdS4cAQzXhYf0dUcw== 2023-10-30T20:22:36.567 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.x] 2023-10-30T20:22:36.567 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBtEEBlFDsIKhAAsIESKg5DUpjYRQGitRBV+Q== 2023-10-30T20:22:36.568 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.4] 2023-10-30T20:22:36.568 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCAEEBlQS9vBBAA4nsgG6Dm08EuGSyAVUilkA== 2023-10-30T20:22:36.568 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.5] 2023-10-30T20:22:36.568 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCDEEBlRmiuAhAAw/BJIJkBMO+Y/yipRPL5Xg== 2023-10-30T20:22:36.568 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.6] 2023-10-30T20:22:36.568 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCGEEBl7lhiABAADMP4utQ7YueoeUiq5us1qg== 2023-10-30T20:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.7] 2023-10-30T20:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCIEEBlON0OOhAAB+uleZf6u7r1r7eF19HANw== 2023-10-30T20:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2023-10-30T20:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBtEEBlcJu8LBAAzgt74k2upEubQTnN8oMlsw== 2023-10-30T20:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout:[client.1] 2023-10-30T20:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBtEEBlYZwiMBAAAnLGPDz4kJiKM6QGXDylnA== 2023-10-30T20:22:36.589 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.y] 2023-10-30T20:22:36.589 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBtEEBls4YpJhAAlhztShxQj/FfHL1kwBA8zQ== 2023-10-30T20:22:36.589 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2023-10-30T20:22:36.589 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBxEEBlhftNBBAA9B+qy/0F2oAh1LIejBSnIQ== 2023-10-30T20:22:36.589 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2023-10-30T20:22:36.590 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQB0EEBlRxVbAhAAvDKnQvBYMEK89xrxDZbFiA== 2023-10-30T20:22:36.590 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2023-10-30T20:22:36.590 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQB3EEBl3mpUABAAfn4Y9gQwt4Ab+O2c0A2McQ== 2023-10-30T20:22:36.590 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.3] 2023-10-30T20:22:36.590 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQB5EEBlBx/yORAAUWI0GBdS4cAQzXhYf0dUcw== 2023-10-30T20:22:36.591 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.x] 2023-10-30T20:22:36.591 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBtEEBlFDsIKhAAsIESKg5DUpjYRQGitRBV+Q== 2023-10-30T20:22:36.591 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.4] 2023-10-30T20:22:36.591 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQCAEEBlQS9vBBAA4nsgG6Dm08EuGSyAVUilkA== 2023-10-30T20:22:36.591 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.5] 2023-10-30T20:22:36.591 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQCDEEBlRmiuAhAAw/BJIJkBMO+Y/yipRPL5Xg== 2023-10-30T20:22:36.592 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.6] 2023-10-30T20:22:36.592 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQCGEEBl7lhiABAADMP4utQ7YueoeUiq5us1qg== 2023-10-30T20:22:36.592 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.7] 2023-10-30T20:22:36.592 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQCIEEBlON0OOhAAB+uleZf6u7r1r7eF19HANw== 2023-10-30T20:22:36.592 INFO:teuthology.orchestra.run.smithi183.stdout:[client.0] 2023-10-30T20:22:36.592 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBtEEBlcJu8LBAAzgt74k2upEubQTnN8oMlsw== 2023-10-30T20:22:36.593 INFO:teuthology.orchestra.run.smithi183.stdout:[client.1] 2023-10-30T20:22:36.593 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBtEEBlYZwiMBAAAnLGPDz4kJiKM6QGXDylnA== 2023-10-30T20:22:36.594 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T20:22:36.614 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T20:22:36.686 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:36.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:36.740 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:36.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:36.794 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:36.796 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:36.848 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:36.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:36.901 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T20:22:36.903 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T20:22:36.955 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:36.957 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:37.008 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:37.011 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:37.061 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:37.064 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:37.115 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:37.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T20:22:37.168 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T20:22:37.171 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T20:22:37.225 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T20:22:37.227 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T20:22:37.281 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-30T20:22:37.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-30T20:22:37.307 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T20:22:37.403 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-30T20:22:37.430 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-30T20:22:37.496 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T20:22:37.592 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-30T20:22:37.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-30T20:22:37.646 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T20:22:37.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-30T20:22:37.769 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T20:22:37.772 DEBUG:teuthology.orchestra.run.smithi183:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T20:22:37.825 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-30T20:22:37.825 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-10-30T20:22:37.825 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T20:22:37.826 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-10-30T20:22:37.828 INFO:tasks.ceph.mon.a:Started 2023-10-30T20:22:37.829 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-10-30T20:22:37.829 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T20:22:37.829 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-10-30T20:22:37.831 INFO:tasks.ceph.mon.c:Started 2023-10-30T20:22:37.832 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-10-30T20:22:37.832 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T20:22:37.832 DEBUG:teuthology.orchestra.run.smithi183:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-10-30T20:22:37.867 INFO:tasks.ceph.mon.b:Started 2023-10-30T20:22:37.868 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-30T20:22:37.868 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-30T20:22:37.868 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-10-30T20:22:37.871 INFO:tasks.ceph.mgr.y:Started 2023-10-30T20:22:37.871 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-30T20:22:37.871 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-10-30T20:22:37.873 INFO:tasks.ceph.mgr.x:Started 2023-10-30T20:22:37.874 INFO:tasks.ceph:Setting crush tunables to default 2023-10-30T20:22:37.874 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd crush tunables default 2023-10-30T20:22:37.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.973+0000 7f6823a7c640 10 monclient: get_monmap_and_config 2023-10-30T20:22:37.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.973+0000 7f6823a7c640 10 monclient: build_initial_monmap 2023-10-30T20:22:37.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.973+0000 7f6823a7c640 10 monclient: monmap: 2023-10-30T20:22:37.975 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:22:37.976 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:22:37.976 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:37.974757+0000 2023-10-30T20:22:37.976 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:37.974757+0000 2023-10-30T20:22:37.976 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:22:37.976 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:22:37.976 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:22:37.977 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:22:37.977 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:22:37.977 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:22:37.977 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:22:37.977 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:22:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.974+0000 7f6823a7c640 1 Processor -- start 2023-10-30T20:22:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.974+0000 7f6823a7c640 1 -- start start 2023-10-30T20:22:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.974+0000 7f6823a7c640 10 monclient: init 2023-10-30T20:22:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6823a7c640 10 monclient: _reopen_session rank -1 2023-10-30T20:22:37.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6823a7c640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2023-10-30T20:22:37.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6823a7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f681c0ae690 0x7f681c070860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:22:37.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6823a7c640 10 monclient(hunting): picked mon.noname-b con 0x7f681c0ae690 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:22:37.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6823a7c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f681c0aea70 addr v1:172.21.15.183:6789/0 2023-10-30T20:22:37.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6823a7c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f681c13cb60 0x7f681c14a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:22:37.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6823a7c640 10 monclient(hunting): picked mon.noname-a con 0x7f681c13cb60 addr v2:172.21.15.69:3300/0 2023-10-30T20:22:37.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6823a7c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c0703d0 con 0x7f681c0aea70 2023-10-30T20:22:37.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6823a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:22:37.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6823a7c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c067b90 con 0x7f681c13cb60 2023-10-30T20:22:37.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6823a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:22:37.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6823a7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0a8700 con 0x7f681c0ae690 2023-10-30T20:22:37.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6823a7c640 10 monclient(hunting): _renew_subs 2023-10-30T20:22:37.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6823a7c640 10 monclient(hunting): authenticate will time out at 1481.264015s 2023-10-30T20:22:37.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6821818640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f681c0ae690 msgr2=0x7f681c070860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2023-10-30T20:22:37.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6821818640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f681c0ae690 0x7f681c070860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:22:37.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6822019640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f681c13cb60 msgr2=0x7f681c14a010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2023-10-30T20:22:37.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6821017640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f681c0aea70 legacy=0x7f681c142c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2023-10-30T20:22:37.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:37.975+0000 7f6822019640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f681c13cb60 0x7f681c14a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:22:38.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:38.176+0000 7f6822019640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f681c13cb60 msgr2=0x7f681c14a010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2023-10-30T20:22:38.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:38.176+0000 7f6821818640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f681c0ae690 msgr2=0x7f681c070860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2023-10-30T20:22:38.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:38.176+0000 7f6822019640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f681c13cb60 0x7f681c14a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:22:38.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:38.176+0000 7f6821818640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f681c0ae690 0x7f681c070860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:22:38.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:38.176+0000 7f6821017640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f681c0aea70 legacy=0x7f681c142c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2023-10-30T20:22:38.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:38.577+0000 7f6821818640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f681c0ae690 msgr2=0x7f681c070860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2023-10-30T20:22:38.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:38.577+0000 7f6822019640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f681c13cb60 msgr2=0x7f681c14a010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2023-10-30T20:22:38.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:38.577+0000 7f6821818640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f681c0ae690 0x7f681c070860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:22:38.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:38.577+0000 7f6822019640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f681c13cb60 0x7f681c14a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:22:38.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:38.577+0000 7f6821017640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f681c0aea70 legacy=0x7f681c142c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2023-10-30T20:22:39.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:39.378+0000 7f6821818640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f681c0ae690 msgr2=0x7f681c070860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2023-10-30T20:22:39.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:39.378+0000 7f6822019640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f681c13cb60 msgr2=0x7f681c14a010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2023-10-30T20:22:39.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:39.378+0000 7f6822019640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f681c13cb60 0x7f681c14a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:22:39.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:39.378+0000 7f6821818640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f681c0ae690 0x7f681c070860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:22:39.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:39.378+0000 7f6821017640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f681c0aea70 legacy=0x7f681c142c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2023-10-30T20:22:40.976 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 10 monclient(hunting): tick 2023-10-30T20:22:40.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T20:22:40.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 1 monclient(hunting): continuing hunt 2023-10-30T20:22:40.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T20:22:40.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f681c0ae690 msgr2=0x7f681c070860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:22:40.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f681c0ae690 0x7f681c070860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:22:40.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f681c13cb60 msgr2=0x7f681c14a010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:22:40.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f681c13cb60 0x7f681c14a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:22:40.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f681c0aea70 legacy=0x7f681c142c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:22:40.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 10 monclient: _add_conns ranks=[4,0] 2023-10-30T20:22:40.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f68000036c0 0x7f6800005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:22:40.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-c con 0x7f68000036c0 addr v2:172.21.15.183:3300/0 2023-10-30T20:22:40.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6800006190 addr v1:172.21.15.69:6789/0 2023-10-30T20:22:40.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c0703d0 con 0x7f6800006190 2023-10-30T20:22:40.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 10 monclient(hunting): start opening mon connection 2023-10-30T20:22:40.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c067b90 con 0x7f68000036c0 2023-10-30T20:22:40.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f68137fe640 10 monclient(hunting): _renew_subs 2023-10-30T20:22:40.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f6821017640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f6800006190 legacy=0x7f6800008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-10-30T20:22:40.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f6821818640 1 -- >> v2:172.21.15.183:3300/0 conn(0x7f68000036c0 msgr2=0x7f6800005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.183:3300/0 2023-10-30T20:22:40.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:40.975+0000 7f6821818640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f68000036c0 0x7f6800005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:22:41.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:41.176+0000 7f6821017640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f6800006190 legacy=0x7f6800008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-10-30T20:22:41.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:41.176+0000 7f6821818640 1 -- >> v2:172.21.15.183:3300/0 conn(0x7f68000036c0 msgr2=0x7f6800005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.183:3300/0 2023-10-30T20:22:41.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:41.176+0000 7f6821818640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f68000036c0 0x7f6800005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:22:41.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:41.577+0000 7f6821017640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f6800006190 legacy=0x7f6800008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-10-30T20:22:41.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:41.577+0000 7f6821818640 1 -- >> v2:172.21.15.183:3300/0 conn(0x7f68000036c0 msgr2=0x7f6800005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.183:3300/0 2023-10-30T20:22:41.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:41.577+0000 7f6821818640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f68000036c0 0x7f6800005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:22:42.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:42.377+0000 7f6821017640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f6800006190 legacy=0x7f6800008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-10-30T20:22:42.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:42.377+0000 7f6821818640 1 -- >> v2:172.21.15.183:3300/0 conn(0x7f68000036c0 msgr2=0x7f6800005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.183:3300/0 2023-10-30T20:22:42.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:42.377+0000 7f6821818640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f68000036c0 0x7f6800005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:22:43.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 10 monclient(hunting): tick 2023-10-30T20:22:43.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T20:22:43.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 1 monclient(hunting): continuing hunt 2023-10-30T20:22:43.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T20:22:43.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 1 -- >> v2:172.21.15.183:3300/0 conn(0x7f68000036c0 msgr2=0x7f6800005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:22:43.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f68000036c0 0x7f6800005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:22:43.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f6800006190 legacy=0x7f6800008650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:22:43.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f6822019640 1 -- reap_dead start 2023-10-30T20:22:43.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-10-30T20:22:43.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800009ef0 0x7f6800005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:22:43.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-b con 0x7f6800009ef0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:22:43.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f680000a370 addr v1:172.21.15.69:6789/0 2023-10-30T20:22:43.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f680000a830 0x7f680000d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:22:43.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-a con 0x7f680000a830 addr v2:172.21.15.69:3300/0 2023-10-30T20:22:43.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c0703d0 con 0x7f680000a370 2023-10-30T20:22:43.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 10 monclient(hunting): start opening mon connection 2023-10-30T20:22:43.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c067b90 con 0x7f680000a830 2023-10-30T20:22:43.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 10 monclient(hunting): start opening mon connection 2023-10-30T20:22:43.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0a8700 con 0x7f6800009ef0 2023-10-30T20:22:43.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f68137fe640 10 monclient(hunting): _renew_subs 2023-10-30T20:22:43.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f6821818640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800009ef0 msgr2=0x7f6800005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2023-10-30T20:22:43.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f6821017640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f680000a370 legacy=0x7f6800008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-10-30T20:22:43.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f6822019640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f680000a830 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2023-10-30T20:22:43.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f6821818640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800009ef0 0x7f6800005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:22:43.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:43.974+0000 7f6822019640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f680000a830 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:22:44.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:44.175+0000 7f6822019640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f680000a830 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2023-10-30T20:22:44.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:44.175+0000 7f6821818640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800009ef0 msgr2=0x7f6800005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2023-10-30T20:22:44.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:44.175+0000 7f6821017640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f680000a370 legacy=0x7f6800008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-10-30T20:22:44.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:44.175+0000 7f6822019640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f680000a830 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:22:44.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:44.175+0000 7f6821818640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800009ef0 0x7f6800005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:22:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:44.576+0000 7f6821818640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800009ef0 msgr2=0x7f6800005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2023-10-30T20:22:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:44.576+0000 7f6821017640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f680000a370 legacy=0x7f6800008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-10-30T20:22:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:44.576+0000 7f6822019640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f680000a830 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2023-10-30T20:22:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:44.576+0000 7f6821818640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800009ef0 0x7f6800005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:22:44.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:44.576+0000 7f6822019640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f680000a830 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:22:45.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:45.377+0000 7f6822019640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f680000a830 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2023-10-30T20:22:45.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:45.377+0000 7f6821818640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800009ef0 msgr2=0x7f6800005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2023-10-30T20:22:45.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:45.377+0000 7f6821017640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f680000a370 legacy=0x7f6800008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-10-30T20:22:45.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:45.377+0000 7f6822019640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f680000a830 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:22:45.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:45.377+0000 7f6821818640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800009ef0 0x7f6800005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:22:45.461 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:00:07.488 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:22:45.772 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:00:07.799 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:22:45.878 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:00:07.937 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:22:45.886 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:00:07.945 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:22:46.204 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:00:08.262 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:22:46.211 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:00:08.270 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:22:46.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 10 monclient(hunting): tick 2023-10-30T20:22:46.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T20:22:46.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 1 monclient(hunting): continuing hunt 2023-10-30T20:22:46.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T20:22:46.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800009ef0 msgr2=0x7f6800005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:22:46.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800009ef0 0x7f6800005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:22:46.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f680000a830 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:22:46.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f680000a830 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:22:46.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f680000a370 legacy=0x7f6800008650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:22:46.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 10 monclient: _add_conns ranks=[4,1] 2023-10-30T20:22:46.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 0x7f680000da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:22:46.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-c con 0x7f680000d5f0 addr v2:172.21.15.183:3300/0 2023-10-30T20:22:46.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f680000e0d0 addr v1:172.21.15.183:6789/0 2023-10-30T20:22:46.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c0703d0 con 0x7f680000e0d0 2023-10-30T20:22:46.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 10 monclient(hunting): start opening mon connection 2023-10-30T20:22:46.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c067b90 con 0x7f680000d5f0 2023-10-30T20:22:46.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:46.975+0000 7f68137fe640 10 monclient(hunting): _renew_subs 2023-10-30T20:22:47.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.248+0000 7f6821017640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f680000e0d0 0x7f6800012790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.69:52866/0 (socket says 172.21.15.69:52866) 2023-10-30T20:22:47.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.248+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 learned_addr learned my addr 172.21.15.69:0/3188780911 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:22:47.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.298+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 0x7f680000da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:47.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.374+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f680000d5f0 auth_method 0 2023-10-30T20:22:47.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.374+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:47.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.374+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:47.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.374+0000 7f6821818640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:22:47.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.425+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v1:172.21.15.183:6789/0 conn(0x7f680000e0d0 legacy=0x7f6800012790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-10-30T20:22:47.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.425+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v1:172.21.15.183:6789/0 conn(0x7f680000e0d0 legacy=0x7f6800012790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T20:22:47.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.425+0000 7f6821017640 1 --1- 172.21.15.69:0/3188780911 >> v1:172.21.15.183:6789/0 conn(0x7f680000e0d0 0x7f6800012790 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2023-10-30T20:22:47.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.425+0000 7f6821017640 1 --1- 172.21.15.69:0/3188780911 >> v1:172.21.15.183:6789/0 conn(0x7f680000e0d0 0x7f6800012790 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2023-10-30T20:22:47.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.425+0000 7f6820816640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.183:6789/0 2023-10-30T20:22:47.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.427+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 msgr2=0x7f680000da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:22:47.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.427+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 msgr2=0x7f680000da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:47.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.427+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 0x7f680000da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:47.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.427+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 0x7f680000da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:22:47.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.629+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 0x7f680000da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:47.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.629+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f680000d5f0 auth_method 0 2023-10-30T20:22:47.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.629+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:47.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.629+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:47.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.629+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:47.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.629+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 msgr2=0x7f680000da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:22:47.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.630+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 msgr2=0x7f680000da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:47.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.630+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 0x7f680000da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:47.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:47.630+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 0x7f680000da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:22:48.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.143+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 0x7f680000da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:48.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.144+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f680000d5f0 auth_method 0 2023-10-30T20:22:48.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.144+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:48.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.144+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:48.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.144+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.153+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 msgr2=0x7f680000da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:22:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.153+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 msgr2=0x7f680000da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.153+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 0x7f680000da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.153+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 0x7f680000da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:22:48.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.954+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 0x7f680000da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:48.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.954+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f680000d5f0 auth_method 0 2023-10-30T20:22:48.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.954+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:48.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.954+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:48.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.954+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:48.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.955+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 msgr2=0x7f680000da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:22:48.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.955+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 msgr2=0x7f680000da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:48.958 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.955+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 0x7f680000da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:48.958 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:48.955+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 0x7f680000da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:22:49.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.975+0000 7f68137fe640 10 monclient(hunting): tick 2023-10-30T20:22:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.975+0000 7f68137fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T20:22:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.975+0000 7f68137fe640 1 monclient(hunting): continuing hunt 2023-10-30T20:22:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.975+0000 7f68137fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T20:22:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.975+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 msgr2=0x7f680000da00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:22:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.975+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000d5f0 0x7f680000da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:22:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.975+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> v1:172.21.15.183:6789/0 conn(0x7f680000e0d0 legacy=0x7f6800012790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T20:22:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.975+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 reap_dead start 2023-10-30T20:22:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.975+0000 7f68137fe640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2023-10-30T20:22:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.975+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6800009ef0 addr v1:172.21.15.69:6789/0 2023-10-30T20:22:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.975+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:22:49.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.975+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-a con 0x7f680000da00 addr v2:172.21.15.69:3300/0 2023-10-30T20:22:49.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.975+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 0x7f68000056c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:22:49.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.976+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-c con 0x7f6800005210 addr v2:172.21.15.183:3300/0 2023-10-30T20:22:49.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.976+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6800005c00 con 0x7f6800009ef0 2023-10-30T20:22:49.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.976+0000 7f68137fe640 10 monclient(hunting): start opening mon connection 2023-10-30T20:22:49.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.976+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c067b90 con 0x7f680000da00 2023-10-30T20:22:49.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.976+0000 7f68137fe640 10 monclient(hunting): start opening mon connection 2023-10-30T20:22:49.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.976+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c0a8700 con 0x7f6800005210 2023-10-30T20:22:49.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.976+0000 7f68137fe640 10 monclient(hunting): _renew_subs 2023-10-30T20:22:49.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.976+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:49.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.976+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 0x7f68000056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:49.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.976+0000 7f6821017640 10 monclient(hunting): get_auth_request con 0x7f680000da00 auth_method 0 2023-10-30T20:22:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.976+0000 7f6821017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.976+0000 7f6821017640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.976+0000 7f6821017640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:22:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.977+0000 7f6822019640 10 monclient(hunting): get_auth_request con 0x7f6800005210 auth_method 0 2023-10-30T20:22:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.977+0000 7f6822019640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:49.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.977+0000 7f6822019640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:49.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.977+0000 7f6822019640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:22:49.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.977+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 msgr2=0x7f68000056c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T20:22:49.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.977+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 msgr2=0x7f68000056c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:49.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.977+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 0x7f68000056c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:49.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.977+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 0x7f68000056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:22:49.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.979+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:22:49.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.979+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:49.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.979+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:49.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:49.979+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:22:50.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.179+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 0x7f68000056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:50.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.179+0000 7f6822019640 10 monclient(hunting): get_auth_request con 0x7f6800005210 auth_method 0 2023-10-30T20:22:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.179+0000 7f6822019640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.179+0000 7f6822019640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.179+0000 7f6822019640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.180+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.180+0000 7f6821017640 10 monclient(hunting): get_auth_request con 0x7f680000da00 auth_method 0 2023-10-30T20:22:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.180+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 msgr2=0x7f68000056c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:22:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.180+0000 7f6821017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.180+0000 7f6821017640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.180+0000 7f6821017640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.180+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 msgr2=0x7f68000056c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.180+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 0x7f68000056c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.181+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 0x7f68000056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:22:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.181+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T20:22:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.181+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:50.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.181+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:50.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.181+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:22:50.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.582+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 0x7f68000056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:50.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.582+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:50.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.583+0000 7f6822019640 10 monclient(hunting): get_auth_request con 0x7f6800005210 auth_method 0 2023-10-30T20:22:50.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.583+0000 7f6822019640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:50.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.583+0000 7f6822019640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:50.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.583+0000 7f6822019640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:50.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.583+0000 7f6821017640 10 monclient(hunting): get_auth_request con 0x7f680000da00 auth_method 0 2023-10-30T20:22:50.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.583+0000 7f6821017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:50.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.583+0000 7f6821017640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:50.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.583+0000 7f6821017640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:50.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.583+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 msgr2=0x7f68000056c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:22:50.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.583+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 msgr2=0x7f68000056c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.583+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 0x7f68000056c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.583+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 0x7f68000056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:22:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.584+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T20:22:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.584+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:50.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.584+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:50.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:50.584+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:22:51.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.385+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 0x7f68000056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:51.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.385+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:51.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.385+0000 7f6822019640 10 monclient(hunting): get_auth_request con 0x7f6800005210 auth_method 0 2023-10-30T20:22:51.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.385+0000 7f6822019640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:51.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.385+0000 7f6822019640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:51.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.385+0000 7f6822019640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:51.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.386+0000 7f6821017640 10 monclient(hunting): get_auth_request con 0x7f680000da00 auth_method 0 2023-10-30T20:22:51.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.386+0000 7f6821017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:51.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.386+0000 7f6821017640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:51.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.386+0000 7f6821017640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:51.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.386+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 msgr2=0x7f68000056c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:22:51.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.386+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 msgr2=0x7f68000056c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:51.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.386+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 0x7f68000056c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:51.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.386+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 0x7f68000056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:22:51.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.386+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T20:22:51.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.386+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:51.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.386+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:51.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:51.386+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:22:52.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 10 monclient(hunting): tick 2023-10-30T20:22:52.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T20:22:52.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 1 monclient(hunting): continuing hunt 2023-10-30T20:22:52.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T20:22:52.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 msgr2=0x7f68000056c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:22:52.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800005210 0x7f68000056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:22:52.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:22:52.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:22:52.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> v1:172.21.15.69:6789/0 conn(0x7f6800009ef0 legacy=0x7f6800003af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:22:52.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 10 monclient: _add_conns ranks=[2,1] 2023-10-30T20:22:52.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 0x7f6800008a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:22:52.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-b con 0x7f6800008650 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:22:52.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6800009000 addr v1:172.21.15.183:6789/0 2023-10-30T20:22:52.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68000094a0 con 0x7f6800009000 2023-10-30T20:22:52.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 10 monclient(hunting): start opening mon connection 2023-10-30T20:22:52.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c067b90 con 0x7f6800008650 2023-10-30T20:22:52.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.976+0000 7f68137fe640 10 monclient(hunting): _renew_subs 2023-10-30T20:22:52.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.977+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 0x7f6800008a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:52.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.977+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f6800008650 auth_method 0 2023-10-30T20:22:52.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.977+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:52.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.977+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:52.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.977+0000 7f6821818640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:22:52.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.978+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 msgr2=0x7f6800008a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:22:52.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.978+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 msgr2=0x7f6800008a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:52.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.978+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 0x7f6800008a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:52.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:52.978+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 0x7f6800008a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:22:53.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.179+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 0x7f6800008a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:53.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.179+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f6800008650 auth_method 0 2023-10-30T20:22:53.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.179+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:53.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.179+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:53.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.179+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:53.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.180+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 msgr2=0x7f6800008a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:22:53.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.180+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 msgr2=0x7f6800008a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:53.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.180+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 0x7f6800008a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:53.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.180+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 0x7f6800008a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:22:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.581+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 0x7f6800008a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.582+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f6800008650 auth_method 0 2023-10-30T20:22:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.582+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:53.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.582+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:53.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.582+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:53.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.582+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 msgr2=0x7f6800008a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:22:53.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.582+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 msgr2=0x7f6800008a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:53.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.582+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 0x7f6800008a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:53.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:53.582+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 0x7f6800008a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:22:54.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:54.384+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 0x7f6800008a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:54.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:54.384+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f6800008650 auth_method 0 2023-10-30T20:22:54.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:54.384+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:54.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:54.384+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:54.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:54.384+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:54.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:54.385+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 msgr2=0x7f6800008a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:22:54.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:54.385+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 msgr2=0x7f6800008a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:54.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:54.385+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 0x7f6800008a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:54.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:54.385+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 0x7f6800008a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:22:55.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 10 monclient(hunting): tick 2023-10-30T20:22:55.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T20:22:55.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 1 monclient(hunting): continuing hunt 2023-10-30T20:22:55.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T20:22:55.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 msgr2=0x7f6800008a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:22:55.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6800008650 0x7f6800008a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:22:55.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> v1:172.21.15.183:6789/0 conn(0x7f6800009000 legacy=0x7f68000147a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:22:55.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 reap_dead start 2023-10-30T20:22:55.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-10-30T20:22:55.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:22:55.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-c con 0x7f680000da00 addr v2:172.21.15.183:3300/0 2023-10-30T20:22:55.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6800009eb0 addr v1:172.21.15.183:6789/0 2023-10-30T20:22:55.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68000056c0 addr v1:172.21.15.69:6789/0 2023-10-30T20:22:55.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6800008d20 con 0x7f68000056c0 2023-10-30T20:22:55.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68000051c0 con 0x7f6800009eb0 2023-10-30T20:22:55.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 10 monclient(hunting): start opening mon connection 2023-10-30T20:22:55.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c067b90 con 0x7f680000da00 2023-10-30T20:22:55.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.976+0000 7f68137fe640 10 monclient(hunting): _renew_subs 2023-10-30T20:22:55.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.978+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:55.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.978+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f680000da00 auth_method 0 2023-10-30T20:22:55.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.978+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:55.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.978+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:55.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.978+0000 7f6821818640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:22:55.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.980+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:22:55.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.980+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:55.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.980+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:55.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:55.980+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:22:56.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.182+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:56.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.182+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f680000da00 auth_method 0 2023-10-30T20:22:56.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.182+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:56.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.182+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:56.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.182+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:56.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.183+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:22:56.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.183+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:56.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.183+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:56.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.183+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:22:56.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.584+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:56.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.585+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f680000da00 auth_method 0 2023-10-30T20:22:56.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.585+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:56.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.585+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:56.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.585+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:56.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.585+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:22:56.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.585+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:56.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.585+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:56.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:56.585+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:22:57.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:57.386+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:57.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:57.387+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f680000da00 auth_method 0 2023-10-30T20:22:57.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:57.387+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:57.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:57.387+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:57.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:57.387+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:57.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:57.387+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:22:57.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:57.387+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:57.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:57.387+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:57.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:57.387+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:22:58.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 10 monclient(hunting): tick 2023-10-30T20:22:58.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T20:22:58.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 1 monclient(hunting): continuing hunt 2023-10-30T20:22:58.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T20:22:58.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:22:58.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f680000da00 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:22:58.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> v1:172.21.15.183:6789/0 conn(0x7f6800009eb0 legacy=0x7f6800008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:22:58.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> v1:172.21.15.69:6789/0 conn(0x7f68000056c0 legacy=0x7f6800004ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:22:58.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 10 monclient: _add_conns ranks=[2,3] 2023-10-30T20:22:58.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 0x7f68000137a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:22:58.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-b con 0x7f68000097b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:22:58.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 0x7f6800014170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:22:58.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-a con 0x7f6800013ce0 addr v2:172.21.15.69:3300/0 2023-10-30T20:22:58.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 10 monclient(hunting): start opening mon connection 2023-10-30T20:22:58.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c067b90 con 0x7f6800013ce0 2023-10-30T20:22:58.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 10 monclient(hunting): start opening mon connection 2023-10-30T20:22:58.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0a8700 con 0x7f68000097b0 2023-10-30T20:22:58.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f68137fe640 10 monclient(hunting): _renew_subs 2023-10-30T20:22:58.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.977+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 0x7f68000137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:58.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.978+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f68000097b0 auth_method 0 2023-10-30T20:22:58.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.978+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:58.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.978+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:58.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.978+0000 7f6821818640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:22:58.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.979+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 0x7f6800014170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:58.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.979+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 msgr2=0x7f68000137a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:22:58.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.979+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 msgr2=0x7f68000137a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:58.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.979+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 0x7f68000137a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:58.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.979+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 0x7f68000137a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:22:58.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.979+0000 7f6821017640 10 monclient(hunting): get_auth_request con 0x7f6800013ce0 auth_method 0 2023-10-30T20:22:58.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.979+0000 7f6821017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:58.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.979+0000 7f6821017640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:58.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.979+0000 7f6821017640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:22:58.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.980+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 msgr2=0x7f6800014170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:22:58.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.980+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 msgr2=0x7f6800014170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:58.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.980+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 0x7f6800014170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:58.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:58.980+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 0x7f6800014170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:22:59.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.180+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 0x7f68000137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:59.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.180+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f68000097b0 auth_method 0 2023-10-30T20:22:59.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.180+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:59.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.180+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:59.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.180+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:59.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.181+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 0x7f6800014170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:59.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.181+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 msgr2=0x7f68000137a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:22:59.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.181+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 msgr2=0x7f68000137a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:59.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.181+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 0x7f68000137a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:59.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.181+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 0x7f68000137a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:22:59.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.181+0000 7f6821017640 10 monclient(hunting): get_auth_request con 0x7f6800013ce0 auth_method 0 2023-10-30T20:22:59.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.181+0000 7f6821017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:59.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.181+0000 7f6821017640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:59.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.181+0000 7f6821017640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:59.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.182+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 msgr2=0x7f6800014170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:22:59.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.182+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 msgr2=0x7f6800014170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:59.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.182+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 0x7f6800014170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:59.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.182+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 0x7f6800014170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:22:59.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.582+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 0x7f68000137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:59.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.583+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f68000097b0 auth_method 0 2023-10-30T20:22:59.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.583+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:59.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.583+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:59.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.583+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:59.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.583+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 0x7f6800014170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:22:59.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.583+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 msgr2=0x7f68000137a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:22:59.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.583+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 msgr2=0x7f68000137a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:59.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.583+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 0x7f68000137a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:59.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.583+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 0x7f68000137a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:22:59.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.584+0000 7f6821017640 10 monclient(hunting): get_auth_request con 0x7f6800013ce0 auth_method 0 2023-10-30T20:22:59.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.584+0000 7f6821017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:22:59.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.584+0000 7f6821017640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:22:59.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.584+0000 7f6821017640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:22:59.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.584+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 msgr2=0x7f6800014170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:22:59.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.584+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 msgr2=0x7f6800014170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:22:59.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.584+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 0x7f6800014170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:22:59.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:22:59.584+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 0x7f6800014170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:23:00.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.385+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 0x7f68000137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:00.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.385+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f68000097b0 auth_method 0 2023-10-30T20:23:00.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.385+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:00.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.385+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:00.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.385+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:23:00.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.386+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 0x7f6800014170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:00.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.386+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 msgr2=0x7f68000137a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:23:00.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.386+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 msgr2=0x7f68000137a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:23:00.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.386+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 0x7f68000137a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:23:00.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.386+0000 7f6821017640 10 monclient(hunting): get_auth_request con 0x7f6800013ce0 auth_method 0 2023-10-30T20:23:00.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.386+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 0x7f68000137a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:23:00.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.386+0000 7f6821017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:00.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.386+0000 7f6821017640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:00.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.386+0000 7f6821017640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:23:00.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.387+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 msgr2=0x7f6800014170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:23:00.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.387+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 msgr2=0x7f6800014170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:23:00.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.387+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 0x7f6800014170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:23:00.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:00.387+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 0x7f6800014170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:23:01.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 10 monclient(hunting): tick 2023-10-30T20:23:01.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T20:23:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 1 monclient(hunting): continuing hunt 2023-10-30T20:23:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T20:23:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 msgr2=0x7f68000137a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:23:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000097b0 0x7f68000137a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:01.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 msgr2=0x7f6800014170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:23:01.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800013ce0 0x7f6800014170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:01.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 reap_dead start 2023-10-30T20:23:01.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-10-30T20:23:01.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f680000d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:01.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-b con 0x7f68000056c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:01.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68000097b0 addr v1:172.21.15.69:6789/0 2023-10-30T20:23:01.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 0x7f6800004ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:01.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-a con 0x7f6800009eb0 addr v2:172.21.15.69:3300/0 2023-10-30T20:23:01.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6800004ff0 con 0x7f68000097b0 2023-10-30T20:23:01.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c067b90 con 0x7f6800009eb0 2023-10-30T20:23:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0a8700 con 0x7f68000056c0 2023-10-30T20:23:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.977+0000 7f68137fe640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.978+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 0x7f6800004ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.978+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f680000d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:01.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.978+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f68000056c0 auth_method 0 2023-10-30T20:23:01.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.978+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:01.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.978+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:01.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.978+0000 7f6821818640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:01.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.978+0000 7f6822019640 10 monclient(hunting): get_auth_request con 0x7f6800009eb0 auth_method 0 2023-10-30T20:23:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.978+0000 7f6822019640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.978+0000 7f6822019640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.978+0000 7f6822019640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.979+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:23:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.979+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:23:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.979+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f680000d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:23:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.979+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:23:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.979+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 msgr2=0x7f6800004ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T20:23:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.979+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 msgr2=0x7f6800004ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:23:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.979+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 0x7f6800004ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:23:01.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:01.979+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 0x7f6800004ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:23:02.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.180+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f680000d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:02.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.180+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 0x7f6800004ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:02.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.180+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f68000056c0 auth_method 0 2023-10-30T20:23:02.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.180+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:02.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.180+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:02.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.180+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:23:02.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.181+0000 7f6822019640 10 monclient(hunting): get_auth_request con 0x7f6800009eb0 auth_method 0 2023-10-30T20:23:02.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.181+0000 7f6822019640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:02.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.181+0000 7f6822019640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:02.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.181+0000 7f6822019640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:23:02.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.181+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:23:02.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.181+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:23:02.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.181+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f680000d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:23:02.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.181+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:23:02.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.181+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 msgr2=0x7f6800004ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T20:23:02.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.182+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 msgr2=0x7f6800004ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:23:02.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.182+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 0x7f6800004ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:23:02.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.182+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 0x7f6800004ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:23:02.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.582+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f680000d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:02.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.583+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f68000056c0 auth_method 0 2023-10-30T20:23:02.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.583+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:02.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.583+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:02.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.583+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:23:02.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.583+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 0x7f6800004ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:02.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.583+0000 7f6822019640 10 monclient(hunting): get_auth_request con 0x7f6800009eb0 auth_method 0 2023-10-30T20:23:02.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.583+0000 7f6822019640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:02.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.583+0000 7f6822019640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:02.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.583+0000 7f6822019640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:23:02.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.584+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:23:02.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.584+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:23:02.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.584+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f680000d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:23:02.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.584+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:23:02.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.584+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 msgr2=0x7f6800004ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T20:23:02.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.584+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 msgr2=0x7f6800004ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:23:02.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.584+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 0x7f6800004ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:23:02.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:02.585+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 0x7f6800004ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:23:03.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.385+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f680000d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:03.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.386+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f68000056c0 auth_method 0 2023-10-30T20:23:03.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.386+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:03.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.386+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:03.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.386+0000 7f6821818640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:23:03.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.386+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 0x7f6800004ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:03.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.386+0000 7f6822019640 10 monclient(hunting): get_auth_request con 0x7f6800009eb0 auth_method 0 2023-10-30T20:23:03.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.386+0000 7f6822019640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:03.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.386+0000 7f6822019640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:03.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.386+0000 7f6822019640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:23:03.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.387+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T20:23:03.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.387+0000 7f6821818640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:23:03.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.387+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f680000d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:23:03.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.387+0000 7f6821818640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:23:03.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.387+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 msgr2=0x7f6800004ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T20:23:03.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.387+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 msgr2=0x7f6800004ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:23:03.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.387+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 0x7f6800004ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:23:03.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:03.387+0000 7f6822019640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 0x7f6800004ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:23:04.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 10 monclient(hunting): tick 2023-10-30T20:23:04.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T20:23:04.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 1 monclient(hunting): continuing hunt 2023-10-30T20:23:04.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T20:23:04.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 msgr2=0x7f680000d060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:23:04.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f680000d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:04.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 msgr2=0x7f6800004ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:23:04.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.69:3300/0 conn(0x7f6800009eb0 0x7f6800004ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:04.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 >> v1:172.21.15.69:6789/0 conn(0x7f68000097b0 legacy=0x7f6800008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:04.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 10 monclient: _add_conns ranks=[1,4] 2023-10-30T20:23:04.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f680000da00 addr v1:172.21.15.183:6789/0 2023-10-30T20:23:04.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 0x7f68000142e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:04.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 10 monclient(hunting): picked mon.noname-c con 0x7f6800013eb0 addr v2:172.21.15.183:3300/0 2023-10-30T20:23:04.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6800014850 con 0x7f680000da00 2023-10-30T20:23:04.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:04.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 1 -- 172.21.15.69:0/3188780911 --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c067b90 con 0x7f6800013eb0 2023-10-30T20:23:04.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f68137fe640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.978+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 0x7f68000142e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.979+0000 7f6821017640 10 monclient(hunting): get_auth_request con 0x7f6800013eb0 auth_method 0 2023-10-30T20:23:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.979+0000 7f6821017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.979+0000 7f6821017640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.979+0000 7f6821017640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.980+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 msgr2=0x7f68000142e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:23:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.980+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 msgr2=0x7f68000142e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:23:04.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.980+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 0x7f68000142e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:23:04.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:04.980+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 0x7f68000142e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:23:05.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.181+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 0x7f68000142e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:05.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.181+0000 7f6821017640 10 monclient(hunting): get_auth_request con 0x7f6800013eb0 auth_method 0 2023-10-30T20:23:05.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.181+0000 7f6821017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:05.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.181+0000 7f6821017640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:05.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.181+0000 7f6821017640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:23:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.183+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 msgr2=0x7f68000142e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:23:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.183+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 msgr2=0x7f68000142e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:23:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.183+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 0x7f68000142e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:23:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.183+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 0x7f68000142e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T20:23:05.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.584+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 0x7f68000142e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:05.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.585+0000 7f6821017640 10 monclient(hunting): get_auth_request con 0x7f6800013eb0 auth_method 0 2023-10-30T20:23:05.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.585+0000 7f6821017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:05.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.585+0000 7f6821017640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:05.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.585+0000 7f6821017640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:23:05.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.586+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 msgr2=0x7f68000142e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:23:05.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.586+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 msgr2=0x7f68000142e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:23:05.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.586+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 0x7f68000142e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:23:05.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:05.586+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 0x7f68000142e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T20:23:06.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:06.387+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 0x7f68000142e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:06.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:06.388+0000 7f6821017640 10 monclient(hunting): get_auth_request con 0x7f6800013eb0 auth_method 0 2023-10-30T20:23:06.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:06.388+0000 7f6821017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:06.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:06.388+0000 7f6821017640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:06.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:06.388+0000 7f6821017640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:23:06.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:06.389+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 msgr2=0x7f68000142e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T20:23:06.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:06.389+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 msgr2=0x7f68000142e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T20:23:06.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:06.389+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 0x7f68000142e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:23:06.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:06.389+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 0x7f68000142e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T20:23:07.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.372+0000 7f6820816640 1 -- 172.21.15.69:0/3188780911 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162239900 0 0) 0x7f68180098c0 con 0x7f680000da00 2023-10-30T20:23:07.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.372+0000 7f6820816640 10 monclient(hunting): _negotiate 2023-10-30T20:23:07.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.372+0000 7f6820816640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:07.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.372+0000 7f6820816640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:07.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.372+0000 7f6820816640 10 monclient(hunting): my global_id is 4102 2023-10-30T20:23:07.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.372+0000 7f6820816640 1 -- 172.21.15.69:0/3188780911 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6808003520 con 0x7f680000da00 2023-10-30T20:23:07.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 1 -- 172.21.15.69:0/3188780911 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 18920562 0 0) 0x7f6818009ba0 con 0x7f680000da00 2023-10-30T20:23:07.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 1 -- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 msgr2=0x7f68000142e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T20:23:07.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 1 --2- 172.21.15.69:0/3188780911 >> v2:172.21.15.183:3300/0 conn(0x7f6800013eb0 0x7f68000142e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:07.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient: _finish_hunting 0 2023-10-30T20:23:07.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 1 monclient: found mon.noname-c-legacy 2023-10-30T20:23:07.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:07.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.183:6789/0 2023-10-30T20:23:07.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 1 -- 172.21.15.69:0/3188780911 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c070ee0 con 0x7f680000da00 2023-10-30T20:23:07.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient: _finish_auth 0 2023-10-30T20:23:07.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient: _check_auth_tickets 2023-10-30T20:23:07.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:07.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 1 -- 172.21.15.69:0/3188780911 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 396370546 0 0) 0x7f6818021040 con 0x7f680000da00 2023-10-30T20:23:07.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:07.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T20:23:07.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient: dump: 2023-10-30T20:23:07.543 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:23:07.543 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:07.543 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:07.543 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:07.543 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:07.544 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:07.544 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:07.544 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:07.544 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:07.544 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:23:07.545 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] but i'm connected to v1:172.21.15.183:6789/0, reconnecting 2023-10-30T20:23:07.545 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:07.545 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 1 -- 172.21.15.69:0/3188780911 >> v1:172.21.15.183:6789/0 conn(0x7f680000da00 legacy=0x7f6800012790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:07.545 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:23:07.545 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6808003970 0x7f6808003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:07.546 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient(hunting): picked mon.a con 0x7f6808003970 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:07.546 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6822019640 1 -- 172.21.15.69:0/3188780911 reap_dead start 2023-10-30T20:23:07.546 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68080047c0 0x7f6808004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:07.546 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient(hunting): picked mon.b con 0x7f68080047c0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:07.546 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f68080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:07.547 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient(hunting): picked mon.c con 0x7f68000056c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:07.547 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:07.547 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 1 -- 172.21.15.69:0/3188780911 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c067b90 con 0x7f6808003970 2023-10-30T20:23:07.547 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:07.547 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 1 -- 172.21.15.69:0/3188780911 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680800bab0 con 0x7f68080047c0 2023-10-30T20:23:07.548 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:07.548 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 1 -- 172.21.15.69:0/3188780911 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680800bc80 con 0x7f68000056c0 2023-10-30T20:23:07.548 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:07.548 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.537+0000 7f6820816640 10 monclient(hunting): _finish_auth 0 2023-10-30T20:23:07.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.544+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68080047c0 0x7f6808004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:07.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.545+0000 7f6821017640 10 monclient(hunting): get_auth_request con 0x7f68080047c0 auth_method 0 2023-10-30T20:23:07.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.545+0000 7f6821017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:07.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.545+0000 7f6821017640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:07.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.545+0000 7f6821017640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:23:07.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.547+0000 7f6821017640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:07.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.547+0000 7f6821017640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:07.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.547+0000 7f6821017640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T20:23:07.553 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.551+0000 7f6821017640 10 monclient(hunting): handle_auth_done global_id 4102 payload 293 2023-10-30T20:23:07.553 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.551+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 msgr2=0x7f68080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:07.554 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.551+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f68080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:07.554 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.552+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6808003970 msgr2=0x7f6808003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:07.554 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.552+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6808003970 0x7f6808003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:07.554 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.552+0000 7f6821017640 10 monclient: _finish_hunting 0 2023-10-30T20:23:07.555 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.552+0000 7f6821017640 1 monclient: found mon.b 2023-10-30T20:23:07.555 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.552+0000 7f6821017640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:07.555 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.552+0000 7f6821017640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:07.555 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.552+0000 7f6821017640 1 -- 172.21.15.69:0/3188780911 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c070ee0 con 0x7f68080047c0 2023-10-30T20:23:07.555 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.552+0000 7f6821017640 10 monclient: _finish_auth 0 2023-10-30T20:23:07.555 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.552+0000 7f6821017640 10 monclient: _check_auth_tickets 2023-10-30T20:23:07.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.552+0000 7f6821017640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:07.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.552+0000 7f6823a7c640 5 monclient: authenticate success, global_id 4102 2023-10-30T20:23:07.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.552+0000 7f6823a7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:23:07.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.555+0000 7f6821017640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68080047c0 0x7f6808004c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f680c000ea0 tx=0x7f68080057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1499c79725964d6e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:07.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.561+0000 7f6820816640 1 -- 172.21.15.69:0/3188780911 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c031050 con 0x7f68080047c0 2023-10-30T20:23:07.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.561+0000 7f6820816640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:07.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.561+0000 7f6820816640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:23:07.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.561+0000 7f6820816640 10 monclient: dump: 2023-10-30T20:23:07.563 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:23:07.564 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:07.564 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:07.564 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:07.564 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:07.564 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:07.565 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:07.565 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:07.565 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:07.565 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:23:07.565 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.561+0000 7f6823a7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:23:07.567 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6820816640 1 -- 172.21.15.69:0/3188780911 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680c020040 con 0x7f68080047c0 2023-10-30T20:23:07.567 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6820816640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:07.567 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6820816640 1 -- 172.21.15.69:0/3188780911 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c00a250 con 0x7f68080047c0 2023-10-30T20:23:07.567 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6823a7c640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:07.567 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6823a7c640 4 set_mon_vals no callback set 2023-10-30T20:23:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6820816640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6820816640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:23:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6820816640 10 monclient: dump: 2023-10-30T20:23:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:23:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:07.569 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:07.569 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:07.569 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:07.569 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:07.569 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:07.569 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:07.570 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:07.570 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:23:07.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6823a7c640 10 monclient: shutdown 2023-10-30T20:23:07.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6823a7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:07.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6823a7c640 1 -- 172.21.15.69:0/3188780911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68080047c0 msgr2=0x7f6808004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:07.571 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6823a7c640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68080047c0 0x7f6808004c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f680c000ea0 tx=0x7f68080057f0 comp rx=0 tx=0).stop 2023-10-30T20:23:07.571 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6823a7c640 1 -- 172.21.15.69:0/3188780911 shutdown_connections 2023-10-30T20:23:07.571 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6823a7c640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68080047c0 0x7f6808004c40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:07.571 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6823a7c640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6808003970 0x7f6808003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:07.571 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6823a7c640 1 --2- 172.21.15.69:0/3188780911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f68080051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:07.572 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.564+0000 7f6823a7c640 1 -- 172.21.15.69:0/3188780911 >> 172.21.15.69:0/3188780911 conn(0x7f681c069d50 msgr2=0x7f681c06a130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:07.572 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.565+0000 7f6823a7c640 1 -- 172.21.15.69:0/3188780911 shutdown_connections 2023-10-30T20:23:07.572 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.565+0000 7f6823a7c640 1 -- 172.21.15.69:0/3188780911 wait complete. 2023-10-30T20:23:07.573 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.566+0000 7f6823a7c640 10 monclient: build_initial_monmap 2023-10-30T20:23:07.573 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.566+0000 7f6823a7c640 10 monclient: monmap: 2023-10-30T20:23:07.574 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:23:07.574 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:07.574 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:23:07.574 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:23:07.575 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:07.575 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:23:07.575 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:07.575 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:07.575 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:07.575 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:23:07.576 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.566+0000 7f6823a7c640 1 librados: starting msgr at 2023-10-30T20:23:07.576 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.566+0000 7f6823a7c640 1 librados: starting objecter 2023-10-30T20:23:07.576 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.566+0000 7f6823a7c640 1 Processor -- start 2023-10-30T20:23:07.576 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.566+0000 7f6823a7c640 1 -- start start 2023-10-30T20:23:07.576 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.566+0000 7f6823a7c640 1 librados: setting wanted keys 2023-10-30T20:23:07.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.566+0000 7f6823a7c640 1 librados: calling monclient init 2023-10-30T20:23:07.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.566+0000 7f6823a7c640 10 monclient: init 2023-10-30T20:23:07.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:07.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T20:23:07.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f681c1d7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:07.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 10 monclient(hunting): picked mon.noname-c con 0x7f68000056c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:07.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6808003970 0x7f681c1d7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:07.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 10 monclient(hunting): picked mon.noname-a con 0x7f6808003970 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:07.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68080047c0 0x7f681c13e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:07.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 10 monclient(hunting): picked mon.noname-b con 0x7f68080047c0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:07.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:07.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0ac440 con 0x7f6808003970 2023-10-30T20:23:07.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:07.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c13d770 con 0x7f68080047c0 2023-10-30T20:23:07.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:07.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0ab770 con 0x7f68000056c0 2023-10-30T20:23:07.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:07.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.567+0000 7f6823a7c640 10 monclient(hunting): authenticate will time out at 1510.856473s 2023-10-30T20:23:07.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.613+0000 7f6822019640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68080047c0 0x7f681c13e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:07.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.623+0000 7f6821818640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f681c1d7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:07.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.623+0000 7f6821818640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f681c1d7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:55116/0 (socket says 172.21.15.69:55116) 2023-10-30T20:23:07.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.623+0000 7f6821818640 1 -- 172.21.15.69:0/2478874596 learned_addr learned my addr 172.21.15.69:0/2478874596 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:23:07.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.623+0000 7f6821818640 10 monclient(hunting): get_auth_request con 0x7f68000056c0 auth_method 0 2023-10-30T20:23:07.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.623+0000 7f6821818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:07.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.623+0000 7f6821818640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:07.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.623+0000 7f6821818640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:07.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.626+0000 7f6821818640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:07.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.626+0000 7f6821818640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:07.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.626+0000 7f6821818640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:07.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.628+0000 7f6822019640 10 monclient(hunting): get_auth_request con 0x7f68080047c0 auth_method 0 2023-10-30T20:23:07.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.628+0000 7f6822019640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:07.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.628+0000 7f6822019640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:07.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.628+0000 7f6822019640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:07.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.632+0000 7f6822019640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:07.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.632+0000 7f6822019640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:07.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.632+0000 7f6822019640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:07.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.635+0000 7f6822019640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2023-10-30T20:23:07.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.635+0000 7f6822019640 1 -- 172.21.15.69:0/2478874596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 msgr2=0x7f681c1d7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.635+0000 7f6822019640 1 --2- 172.21.15.69:0/2478874596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f681c1d7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.635+0000 7f6822019640 1 -- 172.21.15.69:0/2478874596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6808003970 msgr2=0x7f681c1d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.635+0000 7f6822019640 1 --2- 172.21.15.69:0/2478874596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6808003970 0x7f681c1d7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.635+0000 7f6822019640 10 monclient: _finish_hunting 0 2023-10-30T20:23:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.635+0000 7f6822019640 1 monclient: found mon.noname-b 2023-10-30T20:23:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.635+0000 7f6822019640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:07.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.635+0000 7f6822019640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:23:07.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.635+0000 7f6822019640 1 -- 172.21.15.69:0/2478874596 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c139f40 con 0x7f68080047c0 2023-10-30T20:23:07.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.637+0000 7f6822019640 1 --2- 172.21.15.69:0/2478874596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68080047c0 0x7f681c13e420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f681c13cf40 tx=0x7f681400fe40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:07.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.637+0000 7f68127fc640 10 client.?.objecter ms_handle_connect 0x7f68080047c0 2023-10-30T20:23:07.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.637+0000 7f68127fc640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:07.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f68127fc640 1 -- 172.21.15.69:0/2478874596 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681400ba30 con 0x7f68080047c0 2023-10-30T20:23:07.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f68127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:07.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f68127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:23:07.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f68127fc640 10 monclient: dump: 2023-10-30T20:23:07.665 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:23:07.665 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:07.666 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:07.666 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:07.666 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:07.666 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:07.666 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:07.667 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:07.667 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:07.667 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:23:07.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f68127fc640 10 monclient: _finish_auth 0 2023-10-30T20:23:07.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f68127fc640 10 monclient: _check_auth_tickets 2023-10-30T20:23:07.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f68127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:07.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f68127fc640 1 -- 172.21.15.69:0/2478874596 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681400bbc0 con 0x7f68080047c0 2023-10-30T20:23:07.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f6823a7c640 5 monclient: authenticate success, global_id 4108 2023-10-30T20:23:07.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f6823a7c640 10 monclient: _renew_subs 2023-10-30T20:23:07.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f6823a7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:07.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f6823a7c640 1 -- 172.21.15.69:0/2478874596 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c13a1a0 con 0x7f68080047c0 2023-10-30T20:23:07.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f68127fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:07.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f68127fc640 1 -- 172.21.15.69:0/2478874596 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6814019420 con 0x7f68080047c0 2023-10-30T20:23:07.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f68127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:07.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f68127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:23:07.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f68127fc640 10 monclient: dump: 2023-10-30T20:23:07.670 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:23:07.670 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:07.670 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:07.670 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:07.672 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:23:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f6813fff640 4 set_mon_vals no callback set 2023-10-30T20:23:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f6823a7c640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f6823a7c640 10 monclient: _renew_subs 2023-10-30T20:23:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f6823a7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f6823a7c640 1 -- 172.21.15.69:0/2478874596 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c13eb40 con 0x7f68080047c0 2023-10-30T20:23:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.663+0000 7f6823a7c640 1 librados: init done 2023-10-30T20:23:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.664+0000 7f6823a7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:07.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.664+0000 7f6823a7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:07.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.664+0000 7f6823a7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:07.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.664+0000 7f6823a7c640 1 -- 172.21.15.69:0/2478874596 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c1d83f0 con 0x7f68080047c0 2023-10-30T20:23:07.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.897+0000 7f68127fc640 1 -- 172.21.15.69:0/2478874596 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6814019590 con 0x7f68080047c0 2023-10-30T20:23:07.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.903+0000 7f68127fc640 1 -- 172.21.15.69:0/2478874596 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6814023530 con 0x7f68080047c0 2023-10-30T20:23:07.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.903+0000 7f68127fc640 10 client.4108.objecter ms_dispatch 0x7f681c000b90 osd_map(1..1 src has 1..1) v4 2023-10-30T20:23:07.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.903+0000 7f68127fc640 3 client.4108.objecter handle_osd_map got epochs [1,1] > 0 2023-10-30T20:23:07.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.903+0000 7f68127fc640 3 client.4108.objecter handle_osd_map decoding full epoch 1 2023-10-30T20:23:07.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.903+0000 7f68127fc640 20 client.4108.objecter dump_active .. 0 homeless 2023-10-30T20:23:07.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.904+0000 7f68127fc640 1 -- 172.21.15.69:0/2478874596 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f6814019a20 con 0x7f68080047c0 2023-10-30T20:23:07.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.904+0000 7f68127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:07.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.904+0000 7f68127fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.967+0000 7f6823a7c640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T20:23:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.967+0000 7f6823a7c640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T20:23:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.967+0000 7f6823a7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:07.967+0000 7f6823a7c640 1 -- 172.21.15.69:0/2478874596 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f681c0ad160 con 0x7f68080047c0 2023-10-30T20:23:08.099 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:08.098+0000 7fea71613040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T20:23:08.154 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:08.153+0000 7fea71613040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T20:23:08.251 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:08.250+0000 7f594b411040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T20:23:08.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.274+0000 7f68127fc640 1 -- 172.21.15.69:0/2478874596 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6814019bd0 con 0x7f68080047c0 2023-10-30T20:23:08.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.274+0000 7f68127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T20:23:08.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.274+0000 7f68127fc640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-10-30T20:23:08.276 INFO:teuthology.orchestra.run.smithi069.stderr:adjusted tunables profile to default 2023-10-30T20:23:08.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 10 librados: watch_flush enter 2023-10-30T20:23:08.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 10 librados: watch_flush exit 2023-10-30T20:23:08.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 20 client.4108.objecter shutdown clearing up homeless session... 2023-10-30T20:23:08.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 10 client.4108.objecter successfully canceled tick 2023-10-30T20:23:08.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 10 monclient: shutdown 2023-10-30T20:23:08.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:08.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 1 -- 172.21.15.69:0/2478874596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68080047c0 msgr2=0x7f681c13e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:08.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 1 --2- 172.21.15.69:0/2478874596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68080047c0 0x7f681c13e420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f681c13cf40 tx=0x7f681400fe40 comp rx=0 tx=0).stop 2023-10-30T20:23:08.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 1 -- 172.21.15.69:0/2478874596 shutdown_connections 2023-10-30T20:23:08.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 1 --2- 172.21.15.69:0/2478874596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68080047c0 0x7f681c13e420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:08.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 1 --2- 172.21.15.69:0/2478874596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6808003970 0x7f681c1d7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:08.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 1 --2- 172.21.15.69:0/2478874596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68000056c0 0x7f681c1d7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:08.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 1 -- 172.21.15.69:0/2478874596 >> 172.21.15.69:0/2478874596 conn(0x7f681c069d50 msgr2=0x7f681c06a130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:08.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 1 -- 172.21.15.69:0/2478874596 shutdown_connections 2023-10-30T20:23:08.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 1 -- 172.21.15.69:0/2478874596 wait complete. 2023-10-30T20:23:08.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:08.275+0000 7f6823a7c640 1 librados: shutdown 2023-10-30T20:23:08.286 INFO:tasks.ceph:check_enable_crimson: False 2023-10-30T20:23:08.287 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-30T20:23:08.287 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:23:08.288 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-30T20:23:08.304 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:08.303+0000 7f594b411040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T20:23:08.315 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:23:08.316 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-30T20:23:08.384 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:23:08.384 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-30T20:23:08.451 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:23:08.451 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-30T20:23:08.518 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:23:08.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-30T20:23:08.546 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:23:08.546 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-30T20:23:08.612 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:23:08.613 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-30T20:23:08.679 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-10-30T20:23:08.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-30T20:23:08.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new af10bc15-318a-4349-af76-32a30c8acf80 0 2023-10-30T20:23:08.752 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:08.751+0000 7fea71613040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T20:23:08.805 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:08.804+0000 7fea71613040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T20:23:08.849 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:08.848+0000 7fea71613040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T20:23:08.876 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:08.875+0000 7f594b411040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T20:23:08.885 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.883+0000 7ff25eabb640 10 monclient: get_monmap_and_config 2023-10-30T20:23:08.885 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.883+0000 7ff25eabb640 10 monclient: build_initial_monmap 2023-10-30T20:23:08.885 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.884+0000 7ff25eabb640 10 monclient: monmap: 2023-10-30T20:23:08.885 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:08.885 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:08.886 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:23:08.884896+0000 2023-10-30T20:23:08.886 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:23:08.884896+0000 2023-10-30T20:23:08.886 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:08.886 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:08.887 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:08.887 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:08.887 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:08.887 INFO:teuthology.orchestra.run.smithi183.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:08.887 INFO:teuthology.orchestra.run.smithi183.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:08.888 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:08.890 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.884+0000 7ff25eabb640 1 Processor -- start 2023-10-30T20:23:08.890 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.884+0000 7ff25eabb640 1 -- start start 2023-10-30T20:23:08.890 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.884+0000 7ff25eabb640 10 monclient: init 2023-10-30T20:23:08.890 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.885+0000 7ff25eabb640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:08.891 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.885+0000 7ff25eabb640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-10-30T20:23:08.891 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.885+0000 7ff25eabb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff2580b4440 addr v1:172.21.15.183:6789/0 2023-10-30T20:23:08.891 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.885+0000 7ff25eabb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2581455b0 0x7ff25813e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:08.891 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.885+0000 7ff25eabb640 10 monclient(hunting): picked mon.noname-b con 0x7ff2581455b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:08.891 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.885+0000 7ff25eabb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff25813eb80 addr v1:172.21.15.69:6789/0 2023-10-30T20:23:08.892 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.885+0000 7ff25eabb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2580a9da0 con 0x7ff25813eb80 2023-10-30T20:23:08.892 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.885+0000 7ff25eabb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2580aa080 con 0x7ff2580b4440 2023-10-30T20:23:08.892 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.885+0000 7ff25eabb640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:08.892 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.885+0000 7ff25eabb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff258067a10 con 0x7ff2581455b0 2023-10-30T20:23:08.892 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.885+0000 7ff25eabb640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:08.892 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.885+0000 7ff25eabb640 10 monclient(hunting): authenticate will time out at 1508.087404s 2023-10-30T20:23:08.893 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.886+0000 7ff25c857640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff2580b4440 0x7ff258144ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:53844/0 (socket says 172.21.15.183:53844) 2023-10-30T20:23:08.893 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.886+0000 7ff25c857640 1 -- 172.21.15.183:0/2342447010 learned_addr learned my addr 172.21.15.183:0/2342447010 (peer_addr_for_me v1:172.21.15.183:0/0) 2023-10-30T20:23:08.893 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.886+0000 7ff257fff640 1 --2- 172.21.15.183:0/2342447010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2581455b0 0x7ff25813e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:08.893 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.886+0000 7ff257fff640 10 monclient(hunting): get_auth_request con 0x7ff2581455b0 auth_method 0 2023-10-30T20:23:08.894 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.886+0000 7ff257fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:08.894 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.886+0000 7ff257fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:08.894 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.886+0000 7ff257fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:08.894 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.887+0000 7ff257fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:08.894 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.887+0000 7ff257fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:08.894 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.887+0000 7ff257fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:08.895 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.888+0000 7ff2577fe640 1 -- 172.21.15.183:0/2342447010 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600927311 0 0) 0x7ff24c001110 con 0x7ff2580b4440 2023-10-30T20:23:08.895 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.888+0000 7ff2577fe640 10 monclient(hunting): _negotiate 2023-10-30T20:23:08.895 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.888+0000 7ff2577fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:08.895 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.888+0000 7ff2577fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:08.895 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.888+0000 7ff2577fe640 10 monclient(hunting): my global_id is 4111 2023-10-30T20:23:08.896 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.888+0000 7ff2577fe640 1 -- 172.21.15.183:0/2342447010 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff248003520 con 0x7ff2580b4440 2023-10-30T20:23:08.896 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.888+0000 7ff257fff640 10 monclient(hunting): handle_auth_done global_id 4106 payload 274 2023-10-30T20:23:08.896 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.888+0000 7ff257fff640 1 -- 172.21.15.183:0/2342447010 >> v1:172.21.15.183:6789/0 conn(0x7ff2580b4440 legacy=0x7ff258144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:08.896 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.888+0000 7ff257fff640 1 -- 172.21.15.183:0/2342447010 >> v1:172.21.15.69:6789/0 conn(0x7ff25813eb80 legacy=0x7ff25814c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:08.896 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.888+0000 7ff257fff640 10 monclient: _finish_hunting 0 2023-10-30T20:23:08.896 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.888+0000 7ff257fff640 1 monclient: found mon.noname-b 2023-10-30T20:23:08.897 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.888+0000 7ff257fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:08.897 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.888+0000 7ff257fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:23:08.897 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.888+0000 7ff257fff640 1 -- 172.21.15.183:0/2342447010 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2580aa360 con 0x7ff2581455b0 2023-10-30T20:23:08.899 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.890+0000 7ff257fff640 1 --2- 172.21.15.183:0/2342447010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2581455b0 0x7ff25813e640 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff2440097d0 tx=0x7ff24402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53aee6a730977db5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:08.899 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff2577fe640 1 -- 172.21.15.183:0/2342447010 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244031770 con 0x7ff2581455b0 2023-10-30T20:23:08.899 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff2577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:08.899 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff2577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:23:08.899 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff2577fe640 10 monclient: dump: 2023-10-30T20:23:08.900 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:08.900 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:08.900 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:08.900 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:08.900 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:08.901 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:08.901 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:08.901 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:08.901 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:08.901 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:08.901 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff2577fe640 10 monclient: _finish_auth 0 2023-10-30T20:23:08.902 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff2577fe640 10 monclient: _check_auth_tickets 2023-10-30T20:23:08.902 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff2577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:08.902 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff2577fe640 1 -- 172.21.15.183:0/2342447010 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2440318e0 con 0x7ff2581455b0 2023-10-30T20:23:08.902 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff2577fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:08.902 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff2577fe640 1 -- 172.21.15.183:0/2342447010 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244031c80 con 0x7ff2581455b0 2023-10-30T20:23:08.903 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff25eabb640 5 monclient: authenticate success, global_id 4106 2023-10-30T20:23:08.903 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff25eabb640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:08.903 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff25eabb640 4 set_mon_vals no callback set 2023-10-30T20:23:08.903 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff2577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:08.903 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff2577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:23:08.904 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff2577fe640 10 monclient: dump: 2023-10-30T20:23:08.904 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:08.904 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:08.904 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:08.904 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:08.904 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:08.905 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:08.905 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:08.905 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:08.905 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:08.905 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:08.905 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff25eabb640 10 monclient: shutdown 2023-10-30T20:23:08.906 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff25eabb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:08.906 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff25eabb640 1 -- 172.21.15.183:0/2342447010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2581455b0 msgr2=0x7ff25813e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:08.906 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff25eabb640 1 --2- 172.21.15.183:0/2342447010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2581455b0 0x7ff25813e640 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff2440097d0 tx=0x7ff24402b810 comp rx=0 tx=0).stop 2023-10-30T20:23:08.906 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff25eabb640 1 -- 172.21.15.183:0/2342447010 shutdown_connections 2023-10-30T20:23:08.906 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff25eabb640 1 --2- 172.21.15.183:0/2342447010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2581455b0 0x7ff25813e640 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:08.907 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.896+0000 7ff25eabb640 1 -- 172.21.15.183:0/2342447010 >> 172.21.15.183:0/2342447010 conn(0x7ff258065ea0 msgr2=0x7ff2580662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:08.907 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.897+0000 7ff25eabb640 1 -- 172.21.15.183:0/2342447010 shutdown_connections 2023-10-30T20:23:08.907 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.897+0000 7ff25eabb640 1 -- 172.21.15.183:0/2342447010 wait complete. 2023-10-30T20:23:08.907 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.897+0000 7ff25eabb640 10 monclient: build_initial_monmap 2023-10-30T20:23:08.907 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.897+0000 7ff25eabb640 10 monclient: monmap: 2023-10-30T20:23:08.908 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:08.908 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:08.908 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2023-10-30T20:23:08.908 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2023-10-30T20:23:08.908 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:08.908 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2023-10-30T20:23:08.909 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:08.909 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:08.909 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:08.909 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:08.909 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.897+0000 7ff25eabb640 1 librados: starting msgr at 2023-10-30T20:23:08.909 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.897+0000 7ff25eabb640 1 librados: starting objecter 2023-10-30T20:23:08.910 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.897+0000 7ff25eabb640 1 Processor -- start 2023-10-30T20:23:08.910 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 1 -- start start 2023-10-30T20:23:08.910 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 1 librados: setting wanted keys 2023-10-30T20:23:08.910 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 1 librados: calling monclient init 2023-10-30T20:23:08.910 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 10 monclient: init 2023-10-30T20:23:08.911 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:08.902+0000 7fea71613040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T20:23:08.915 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:08.915 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T20:23:08.915 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2580b4440 0x7ff2581d71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:08.915 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 10 monclient(hunting): picked mon.noname-c con 0x7ff2580b4440 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:08.915 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff25813eb80 0x7ff2581d7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:08.916 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 10 monclient(hunting): picked mon.noname-a con 0x7ff25813eb80 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:08.916 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2581455b0 0x7ff2580af470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:08.916 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 10 monclient(hunting): picked mon.noname-b con 0x7ff2581455b0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:08.916 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:08.917 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580a8c70 con 0x7ff25813eb80 2023-10-30T20:23:08.917 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:08.917 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580aefe0 con 0x7ff2581455b0 2023-10-30T20:23:08.917 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:08.917 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2581d7c50 con 0x7ff2580b4440 2023-10-30T20:23:08.917 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:08.918 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.898+0000 7ff25eabb640 10 monclient(hunting): authenticate will time out at 1508.100731s 2023-10-30T20:23:08.918 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.899+0000 7ff25d058640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2581455b0 0x7ff2580af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:08.918 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.899+0000 7ff257fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff25813eb80 0x7ff2581d7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:08.918 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.899+0000 7ff25d058640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2581455b0 0x7ff2580af470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:58368/0 (socket says 172.21.15.183:58368) 2023-10-30T20:23:08.918 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.899+0000 7ff25d058640 1 -- 172.21.15.183:0/867977840 learned_addr learned my addr 172.21.15.183:0/867977840 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-10-30T20:23:08.919 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.899+0000 7ff25d058640 10 monclient(hunting): get_auth_request con 0x7ff2581455b0 auth_method 0 2023-10-30T20:23:08.919 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.899+0000 7ff25d058640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:08.919 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.899+0000 7ff25d058640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:08.919 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.899+0000 7ff25d058640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:08.919 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.899+0000 7ff257fff640 10 monclient(hunting): get_auth_request con 0x7ff25813eb80 auth_method 0 2023-10-30T20:23:08.920 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.899+0000 7ff257fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:08.920 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.899+0000 7ff257fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:08.920 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.899+0000 7ff257fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:08.920 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.900+0000 7ff25d058640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:08.920 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.900+0000 7ff25d058640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:08.920 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.900+0000 7ff25d058640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:08.921 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.900+0000 7ff257fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:08.921 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.900+0000 7ff257fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:08.921 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.900+0000 7ff257fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:08.921 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.902+0000 7ff25d058640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2023-10-30T20:23:08.921 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.902+0000 7ff25c857640 1 --2- 172.21.15.183:0/867977840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2580b4440 0x7ff2581d71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:08.922 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.902+0000 7ff25d058640 1 -- 172.21.15.183:0/867977840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2580b4440 msgr2=0x7ff2581d71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:08.922 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.902+0000 7ff25d058640 1 --2- 172.21.15.183:0/867977840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2580b4440 0x7ff2581d71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:08.922 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.902+0000 7ff25d058640 1 -- 172.21.15.183:0/867977840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff25813eb80 msgr2=0x7ff2581d7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:08.922 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.902+0000 7ff25d058640 1 --2- 172.21.15.183:0/867977840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff25813eb80 0x7ff2581d7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:08.923 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.902+0000 7ff25d058640 10 monclient: _finish_hunting 0 2023-10-30T20:23:08.923 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.902+0000 7ff25d058640 1 monclient: found mon.noname-b 2023-10-30T20:23:08.923 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.902+0000 7ff25d058640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:08.923 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.903+0000 7ff25d058640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:23:08.923 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.903+0000 7ff25d058640 1 -- 172.21.15.183:0/867977840 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2581ce330 con 0x7ff2581455b0 2023-10-30T20:23:08.924 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.905+0000 7ff25d058640 1 --2- 172.21.15.183:0/867977840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2581455b0 0x7ff2580af470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff25000c7d0 tx=0x7ff25000cc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:08.924 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.905+0000 7ff2557fa640 10 client.?.objecter ms_handle_connect 0x7ff2581455b0 2023-10-30T20:23:08.924 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.905+0000 7ff2557fa640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:08.924 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff2557fa640 1 -- 172.21.15.183:0/867977840 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff250003b70 con 0x7ff2581455b0 2023-10-30T20:23:08.924 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff2557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:08.925 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff2557fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:23:08.925 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff2557fa640 10 monclient: dump: 2023-10-30T20:23:08.925 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:08.925 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:08.925 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:08.926 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:08.926 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:08.926 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:08.926 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:08.926 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:08.926 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:08.927 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:08.927 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff2557fa640 10 monclient: _finish_auth 0 2023-10-30T20:23:08.927 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff2557fa640 10 monclient: _check_auth_tickets 2023-10-30T20:23:08.927 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff2557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:08.927 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff2557fa640 1 -- 172.21.15.183:0/867977840 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25000e040 con 0x7ff2581455b0 2023-10-30T20:23:08.928 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff2557fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:08.928 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff2557fa640 1 -- 172.21.15.183:0/867977840 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25000a850 con 0x7ff2581455b0 2023-10-30T20:23:08.928 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff2557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:08.928 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff2557fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:23:08.928 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff2557fa640 10 monclient: dump: 2023-10-30T20:23:08.928 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:08.929 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:08.929 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:08.929 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:08.929 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:08.929 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:08.929 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:08.930 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:08.930 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:08.930 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:08.930 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff25eabb640 5 monclient: authenticate success, global_id 4114 2023-10-30T20:23:08.930 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff25eabb640 10 monclient: _renew_subs 2023-10-30T20:23:08.931 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff25eabb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:08.931 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff256ffd640 4 set_mon_vals no callback set 2023-10-30T20:23:08.931 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff25eabb640 1 -- 172.21.15.183:0/867977840 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2580af9b0 con 0x7ff2581455b0 2023-10-30T20:23:08.931 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff25eabb640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:08.931 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff25eabb640 10 monclient: _renew_subs 2023-10-30T20:23:08.931 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff25eabb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:08.932 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff25eabb640 1 -- 172.21.15.183:0/867977840 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2580afe50 con 0x7ff2581455b0 2023-10-30T20:23:08.932 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.908+0000 7ff25eabb640 1 librados: init done 2023-10-30T20:23:08.932 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.909+0000 7ff25eabb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:08.932 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.909+0000 7ff25eabb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:08.932 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.909+0000 7ff25eabb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:08.933 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.909+0000 7ff25eabb640 1 -- 172.21.15.183:0/867977840 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2580af190 con 0x7ff2581455b0 2023-10-30T20:23:08.933 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.912+0000 7ff2557fa640 1 -- 172.21.15.183:0/867977840 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff25000acc0 con 0x7ff2581455b0 2023-10-30T20:23:08.933 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.912+0000 7ff2557fa640 1 -- 172.21.15.183:0/867977840 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff250025790 con 0x7ff2581455b0 2023-10-30T20:23:08.933 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.912+0000 7ff2557fa640 10 client.4114.objecter ms_dispatch 0x7ff258000b90 osd_map(2..2 src has 1..2) v4 2023-10-30T20:23:08.934 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.912+0000 7ff2557fa640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2023-10-30T20:23:08.934 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.912+0000 7ff2557fa640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2023-10-30T20:23:08.934 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:08.912+0000 7ff2557fa640 20 client.4114.objecter dump_active .. 0 homeless 2023-10-30T20:23:08.934 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:08.931+0000 7f594b411040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T20:23:08.946 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:08.945+0000 7fea71613040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T20:23:08.977 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:08.975+0000 7f594b411040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T20:23:09.007 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.006+0000 7ff2557fa640 1 -- 172.21.15.183:0/867977840 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7ff2500057b0 con 0x7ff2581455b0 2023-10-30T20:23:09.007 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.006+0000 7ff2557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:09.008 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.006+0000 7ff2557fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:09.032 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:09.031+0000 7f594b411040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T20:23:09.045 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:09.044+0000 7fea71613040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-30T20:23:09.075 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:09.073+0000 7f594b411040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T20:23:09.079 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.077+0000 7ff25eabb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "af10bc15-318a-4349-af76-32a30c8acf80", "id": 0}] 2023-10-30T20:23:09.079 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.077+0000 7ff25eabb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "af10bc15-318a-4349-af76-32a30c8acf80", "id": 0}] 2023-10-30T20:23:09.079 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.077+0000 7ff25eabb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:09.079 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.077+0000 7ff25eabb640 1 -- 172.21.15.183:0/867977840 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd new", "uuid": "af10bc15-318a-4349-af76-32a30c8acf80", "id": 0} v 0) v1 -- 0x7ff2581d7e20 con 0x7ff2581455b0 2023-10-30T20:23:09.114 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:09.113+0000 7fea71613040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T20:23:09.162 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:09.161+0000 7fea71613040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T20:23:09.177 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:09.176+0000 7f594b411040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-30T20:23:09.221 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:09.219+0000 7fea71613040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T20:23:09.234 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.233+0000 7ff2557fa640 1 -- 172.21.15.183:0/867977840 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "af10bc15-318a-4349-af76-32a30c8acf80", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff2500050d0 con 0x7ff2581455b0 2023-10-30T20:23:09.235 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.233+0000 7ff2557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "af10bc15-318a-4349-af76-32a30c8acf80", "id": 0}] 2023-10-30T20:23:09.235 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.233+0000 7ff2557fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:09.235 INFO:teuthology.orchestra.run.smithi183.stdout:0 2023-10-30T20:23:09.237 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 10 librados: watch_flush enter 2023-10-30T20:23:09.237 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 10 librados: watch_flush exit 2023-10-30T20:23:09.237 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 20 client.4114.objecter shutdown clearing up homeless session... 2023-10-30T20:23:09.237 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 10 client.4114.objecter successfully canceled tick 2023-10-30T20:23:09.237 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 10 monclient: shutdown 2023-10-30T20:23:09.238 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:09.238 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 1 -- 172.21.15.183:0/867977840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2581455b0 msgr2=0x7ff2580af470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:09.238 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 1 --2- 172.21.15.183:0/867977840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2581455b0 0x7ff2580af470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff25000c7d0 tx=0x7ff25000cc90 comp rx=0 tx=0).stop 2023-10-30T20:23:09.238 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 1 -- 172.21.15.183:0/867977840 shutdown_connections 2023-10-30T20:23:09.238 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 1 --2- 172.21.15.183:0/867977840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2581455b0 0x7ff2580af470 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.238 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 1 --2- 172.21.15.183:0/867977840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff25813eb80 0x7ff2581d7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.239 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 1 --2- 172.21.15.183:0/867977840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2580b4440 0x7ff2581d71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.239 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 1 -- 172.21.15.183:0/867977840 >> 172.21.15.183:0/867977840 conn(0x7ff258065ea0 msgr2=0x7ff25813cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:09.239 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 1 -- 172.21.15.183:0/867977840 shutdown_connections 2023-10-30T20:23:09.239 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 1 -- 172.21.15.183:0/867977840 wait complete. 2023-10-30T20:23:09.239 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.234+0000 7ff25eabb640 1 librados: shutdown 2023-10-30T20:23:09.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new d0d32a9a-e24a-4d0c-9b90-125485d70672 1 2023-10-30T20:23:09.252 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:09.251+0000 7f594b411040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T20:23:09.269 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:09.268+0000 7fea71613040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T20:23:09.300 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:09.299+0000 7f594b411040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T20:23:09.348 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.347+0000 7f5bff461640 10 monclient: get_monmap_and_config 2023-10-30T20:23:09.349 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.347+0000 7f5bff461640 10 monclient: build_initial_monmap 2023-10-30T20:23:09.349 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.347+0000 7f5bff461640 10 monclient: monmap: 2023-10-30T20:23:09.349 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:09.349 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:09.349 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:23:09.348493+0000 2023-10-30T20:23:09.350 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:23:09.348493+0000 2023-10-30T20:23:09.350 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:09.350 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:09.350 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:09.351 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:09.351 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:09.351 INFO:teuthology.orchestra.run.smithi183.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:09.351 INFO:teuthology.orchestra.run.smithi183.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:09.351 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:09.354 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.348+0000 7f5bff461640 1 Processor -- start 2023-10-30T20:23:09.354 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.348+0000 7f5bff461640 1 -- start start 2023-10-30T20:23:09.354 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.348+0000 7f5bff461640 10 monclient: init 2023-10-30T20:23:09.354 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.348+0000 7f5bff461640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:09.355 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.348+0000 7f5bff461640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2023-10-30T20:23:09.355 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.348+0000 7f5bff461640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5bf8065ab0 addr v1:172.21.15.69:6789/0 2023-10-30T20:23:09.355 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.348+0000 7f5bff461640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5bf8065e90 0x7f5bf80ac9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:09.355 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.348+0000 7f5bff461640 10 monclient(hunting): picked mon.noname-a con 0x7f5bf8065e90 addr v2:172.21.15.69:3300/0 2023-10-30T20:23:09.355 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.348+0000 7f5bff461640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5bf80acf20 0x7f5bf814c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:09.355 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.348+0000 7f5bff461640 10 monclient(hunting): picked mon.noname-c con 0x7f5bf80acf20 addr v2:172.21.15.183:3300/0 2023-10-30T20:23:09.356 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bff461640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf80aec80 con 0x7f5bf8065ab0 2023-10-30T20:23:09.356 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bff461640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:09.356 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bff461640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf8074040 con 0x7f5bf8065e90 2023-10-30T20:23:09.356 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bff461640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:09.356 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bff461640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf806e3d0 con 0x7f5bf80acf20 2023-10-30T20:23:09.357 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bff461640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:09.357 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bff461640 10 monclient(hunting): authenticate will time out at 1508.551056s 2023-10-30T20:23:09.357 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bfd9fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5bf80acf20 0x7f5bf814c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:09.357 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bfc9fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5bf8065e90 0x7f5bf80ac9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:09.357 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bfd9fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5bf80acf20 0x7f5bf814c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:58370/0 (socket says 172.21.15.183:58370) 2023-10-30T20:23:09.358 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bfd9fe640 1 -- 172.21.15.183:0/517088491 learned_addr learned my addr 172.21.15.183:0/517088491 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-10-30T20:23:09.358 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bfd9fe640 10 monclient(hunting): get_auth_request con 0x7f5bf80acf20 auth_method 0 2023-10-30T20:23:09.358 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bfd9fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:09.358 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bfd9fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:09.358 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bfd9fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:09.359 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bfc9fc640 10 monclient(hunting): get_auth_request con 0x7f5bf8065e90 auth_method 0 2023-10-30T20:23:09.359 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bfc9fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:09.359 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bfc9fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:09.359 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.349+0000 7f5bfc9fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:09.359 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.350+0000 7f5bfd9fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:09.360 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.350+0000 7f5bfd9fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:09.360 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.350+0000 7f5bfd9fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:09.360 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.350+0000 7f5bfc9fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:09.360 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.350+0000 7f5bfc9fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:09.360 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.350+0000 7f5bfc9fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:09.361 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.352+0000 7f5bfd9fe640 10 monclient(hunting): handle_auth_done global_id 4117 payload 274 2023-10-30T20:23:09.361 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.352+0000 7f5bfd9fe640 1 -- 172.21.15.183:0/517088491 >> v2:172.21.15.69:3300/0 conn(0x7f5bf8065e90 msgr2=0x7f5bf80ac9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:09.361 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.352+0000 7f5bfd9fe640 1 --2- 172.21.15.183:0/517088491 >> v2:172.21.15.69:3300/0 conn(0x7f5bf8065e90 0x7f5bf80ac9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.361 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.352+0000 7f5bfd9fe640 1 -- 172.21.15.183:0/517088491 >> v1:172.21.15.69:6789/0 conn(0x7f5bf8065ab0 legacy=0x7f5bf8070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:09.362 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.352+0000 7f5bfd9fe640 10 monclient: _finish_hunting 0 2023-10-30T20:23:09.362 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.352+0000 7f5bfd9fe640 1 monclient: found mon.noname-c 2023-10-30T20:23:09.362 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.352+0000 7f5bfd9fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:09.362 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.352+0000 7f5bfd9fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:23:09.363 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.352+0000 7f5bfd9fe640 1 -- 172.21.15.183:0/517088491 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf80af120 con 0x7f5bf80acf20 2023-10-30T20:23:09.363 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.353+0000 7f5bfd9fe640 1 --2- 172.21.15.183:0/517088491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bf80acf20 0x7f5bf814c330 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5bf80ae200 tx=0x7f5bf002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5a9560589fa6535 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:09.364 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:09.356+0000 7f594b411040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T20:23:09.367 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5beffff640 1 -- 172.21.15.183:0/517088491 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf002ca80 con 0x7f5bf80acf20 2023-10-30T20:23:09.367 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5beffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:09.367 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5beffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:23:09.367 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5beffff640 10 monclient: dump: 2023-10-30T20:23:09.367 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:09.368 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:09.368 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.368 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.368 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:09.368 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:09.369 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:09.369 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:09.369 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:09.369 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:09.369 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5beffff640 10 monclient: _finish_auth 0 2023-10-30T20:23:09.369 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5beffff640 10 monclient: _check_auth_tickets 2023-10-30T20:23:09.370 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5beffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:09.370 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5beffff640 1 -- 172.21.15.183:0/517088491 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf002cbf0 con 0x7f5bf80acf20 2023-10-30T20:23:09.370 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5bff461640 5 monclient: authenticate success, global_id 4117 2023-10-30T20:23:09.370 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5bff461640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:23:09.370 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5beffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:09.370 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5bff461640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:09.371 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5bff461640 4 set_mon_vals no callback set 2023-10-30T20:23:09.371 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5beffff640 1 -- 172.21.15.183:0/517088491 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf00343f0 con 0x7f5bf80acf20 2023-10-30T20:23:09.371 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5beffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:09.371 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5beffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:23:09.371 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.358+0000 7f5beffff640 10 monclient: dump: 2023-10-30T20:23:09.372 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:09.372 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:09.372 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.372 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.373 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:09.373 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:09.373 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:09.373 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:09.373 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:09.373 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:09.374 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.359+0000 7f5bff461640 10 monclient: shutdown 2023-10-30T20:23:09.374 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.359+0000 7f5bff461640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:09.374 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.359+0000 7f5bff461640 1 -- 172.21.15.183:0/517088491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bf80acf20 msgr2=0x7f5bf814c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:09.374 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.359+0000 7f5bff461640 1 --2- 172.21.15.183:0/517088491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bf80acf20 0x7f5bf814c330 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5bf80ae200 tx=0x7f5bf002bee0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.374 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.359+0000 7f5bff461640 1 -- 172.21.15.183:0/517088491 shutdown_connections 2023-10-30T20:23:09.375 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.359+0000 7f5bff461640 1 --2- 172.21.15.183:0/517088491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bf80acf20 0x7f5bf814c330 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.375 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.359+0000 7f5bff461640 1 --2- 172.21.15.183:0/517088491 >> v2:172.21.15.69:3300/0 conn(0x7f5bf8065e90 0x7f5bf80ac9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.375 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.359+0000 7f5bff461640 1 -- 172.21.15.183:0/517088491 >> 172.21.15.183:0/517088491 conn(0x7f5bf80b3820 msgr2=0x7f5bf80b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:09.375 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.359+0000 7f5bff461640 1 -- 172.21.15.183:0/517088491 shutdown_connections 2023-10-30T20:23:09.375 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.359+0000 7f5bff461640 1 -- 172.21.15.183:0/517088491 wait complete. 2023-10-30T20:23:09.376 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.360+0000 7f5bff461640 10 monclient: build_initial_monmap 2023-10-30T20:23:09.376 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.360+0000 7f5bff461640 10 monclient: monmap: 2023-10-30T20:23:09.376 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:09.376 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:09.376 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2023-10-30T20:23:09.377 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2023-10-30T20:23:09.377 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:09.377 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2023-10-30T20:23:09.377 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:09.377 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:09.377 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:09.378 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:09.378 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.360+0000 7f5bff461640 1 librados: starting msgr at 2023-10-30T20:23:09.378 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.360+0000 7f5bff461640 1 librados: starting objecter 2023-10-30T20:23:09.378 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.360+0000 7f5bff461640 1 Processor -- start 2023-10-30T20:23:09.378 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.360+0000 7f5bff461640 1 -- start start 2023-10-30T20:23:09.378 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.360+0000 7f5bff461640 1 librados: setting wanted keys 2023-10-30T20:23:09.379 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.360+0000 7f5bff461640 1 librados: calling monclient init 2023-10-30T20:23:09.379 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.360+0000 7f5bff461640 10 monclient: init 2023-10-30T20:23:09.379 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:09.379 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T20:23:09.379 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf8065ab0 0x7f5bf81cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:09.380 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 10 monclient(hunting): picked mon.noname-a con 0x7f5bf8065ab0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:09.380 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf80acf20 0x7f5bf81d6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:09.380 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 10 monclient(hunting): picked mon.noname-c con 0x7f5bf80acf20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:09.380 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bf81d64e0 0x7f5bf813f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:09.380 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 10 monclient(hunting): picked mon.noname-b con 0x7f5bf81d64e0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:09.380 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:09.381 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf80b0f00 con 0x7f5bf8065ab0 2023-10-30T20:23:09.381 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:09.381 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf8138bd0 con 0x7f5bf81d64e0 2023-10-30T20:23:09.381 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:09.381 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf81d6ae0 con 0x7f5bf80acf20 2023-10-30T20:23:09.382 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:09.382 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bff461640 10 monclient(hunting): authenticate will time out at 1508.563409s 2023-10-30T20:23:09.382 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bfd9fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bf81d64e0 0x7f5bf813f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:09.382 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.361+0000 7f5bfd1fd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf8065ab0 0x7f5bf81cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:09.382 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfc9fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf80acf20 0x7f5bf81d6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:09.383 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfd9fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bf81d64e0 0x7f5bf813f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:58376/0 (socket says 172.21.15.183:58376) 2023-10-30T20:23:09.383 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfd9fe640 1 -- 172.21.15.183:0/2865146696 learned_addr learned my addr 172.21.15.183:0/2865146696 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-10-30T20:23:09.383 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfd9fe640 10 monclient(hunting): get_auth_request con 0x7f5bf81d64e0 auth_method 0 2023-10-30T20:23:09.383 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfd9fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:09.383 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfd9fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:09.384 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfd9fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:09.384 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfd1fd640 10 monclient(hunting): get_auth_request con 0x7f5bf8065ab0 auth_method 0 2023-10-30T20:23:09.384 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfd1fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:09.384 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfd1fd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:09.384 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfd1fd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:09.384 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfc9fc640 10 monclient(hunting): get_auth_request con 0x7f5bf80acf20 auth_method 0 2023-10-30T20:23:09.385 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfc9fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:09.385 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfc9fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:09.385 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfc9fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:09.385 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfd9fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:09.385 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfd9fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:09.385 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.362+0000 7f5bfd9fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:09.386 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.363+0000 7f5bfd1fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:09.386 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.363+0000 7f5bfd1fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:09.386 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.363+0000 7f5bfd1fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:09.386 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.363+0000 7f5bfc9fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:09.386 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.363+0000 7f5bfc9fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:09.387 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.363+0000 7f5bfc9fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:09.387 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.365+0000 7f5bfd9fe640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2023-10-30T20:23:09.387 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.365+0000 7f5bfd9fe640 1 -- 172.21.15.183:0/2865146696 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf80acf20 msgr2=0x7f5bf81d6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:09.387 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.365+0000 7f5bfd9fe640 1 --2- 172.21.15.183:0/2865146696 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf80acf20 0x7f5bf81d6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.387 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.365+0000 7f5bfd9fe640 1 -- 172.21.15.183:0/2865146696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf8065ab0 msgr2=0x7f5bf81cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:09.388 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.365+0000 7f5bfd9fe640 1 --2- 172.21.15.183:0/2865146696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf8065ab0 0x7f5bf81cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.388 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.365+0000 7f5bfd9fe640 10 monclient: _finish_hunting 0 2023-10-30T20:23:09.388 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.365+0000 7f5bfd9fe640 1 monclient: found mon.noname-b 2023-10-30T20:23:09.388 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.365+0000 7f5bfd9fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:09.388 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.365+0000 7f5bfd9fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:23:09.389 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.365+0000 7f5bfd9fe640 1 -- 172.21.15.183:0/2865146696 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf813ec40 con 0x7f5bf81d64e0 2023-10-30T20:23:09.390 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.367+0000 7f5bfd9fe640 1 --2- 172.21.15.183:0/2865146696 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bf81d64e0 0x7f5bf813f820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5bf002c4f0 tx=0x7f5bf0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:09.391 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.367+0000 7f5bedffb640 10 client.?.objecter ms_handle_connect 0x7f5bf81d64e0 2023-10-30T20:23:09.391 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.367+0000 7f5bedffb640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:09.391 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bedffb640 1 -- 172.21.15.183:0/2865146696 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf003e450 con 0x7f5bf81d64e0 2023-10-30T20:23:09.391 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:09.392 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bedffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:23:09.392 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bedffb640 10 monclient: dump: 2023-10-30T20:23:09.392 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:09.392 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:09.393 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.393 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.393 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:09.393 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:09.393 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:09.393 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:09.394 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:09.394 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:09.394 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bedffb640 10 monclient: _finish_auth 0 2023-10-30T20:23:09.394 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bedffb640 10 monclient: _check_auth_tickets 2023-10-30T20:23:09.394 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:09.395 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bedffb640 1 -- 172.21.15.183:0/2865146696 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf003e5e0 con 0x7f5bf81d64e0 2023-10-30T20:23:09.395 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bff461640 5 monclient: authenticate success, global_id 4120 2023-10-30T20:23:09.395 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bff461640 10 monclient: _renew_subs 2023-10-30T20:23:09.395 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bff461640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:09.395 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bff461640 1 -- 172.21.15.183:0/2865146696 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf813fd60 con 0x7f5bf81d64e0 2023-10-30T20:23:09.396 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bedffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:09.396 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bedffb640 1 -- 172.21.15.183:0/2865146696 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf003e9a0 con 0x7f5bf81d64e0 2023-10-30T20:23:09.396 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:09.396 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bedffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:23:09.396 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bedffb640 10 monclient: dump: 2023-10-30T20:23:09.396 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:09.397 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:09.397 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.397 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.397 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:09.397 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:09.397 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:09.398 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:09.398 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:09.398 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:09.398 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.370+0000 7f5bef7fe640 4 set_mon_vals no callback set 2023-10-30T20:23:09.398 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.371+0000 7f5bff461640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:09.399 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.371+0000 7f5bff461640 10 monclient: _renew_subs 2023-10-30T20:23:09.399 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.371+0000 7f5bff461640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:09.399 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.371+0000 7f5bff461640 1 -- 172.21.15.183:0/2865146696 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf813e310 con 0x7f5bf81d64e0 2023-10-30T20:23:09.399 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.371+0000 7f5bff461640 1 librados: init done 2023-10-30T20:23:09.399 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.372+0000 7f5bff461640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:09.400 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.372+0000 7f5bff461640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:09.400 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.372+0000 7f5bff461640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:09.400 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.372+0000 7f5bff461640 1 -- 172.21.15.183:0/2865146696 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf81d6c30 con 0x7f5bf81d64e0 2023-10-30T20:23:09.400 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.374+0000 7f5bedffb640 1 -- 172.21.15.183:0/2865146696 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5bf002ca90 con 0x7f5bf81d64e0 2023-10-30T20:23:09.400 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.374+0000 7f5bedffb640 1 -- 172.21.15.183:0/2865146696 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f5bf004c030 con 0x7f5bf81d64e0 2023-10-30T20:23:09.400 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.374+0000 7f5bedffb640 10 client.4120.objecter ms_dispatch 0x7f5bf8000b90 osd_map(3..3 src has 1..3) v4 2023-10-30T20:23:09.401 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.374+0000 7f5bedffb640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2023-10-30T20:23:09.401 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.374+0000 7f5bedffb640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2023-10-30T20:23:09.401 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.374+0000 7f5bedffb640 20 client.4120.objecter dump_active .. 0 homeless 2023-10-30T20:23:09.406 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:09.405+0000 7f594b411040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T20:23:09.469 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.468+0000 7f5bedffb640 1 -- 172.21.15.183:0/2865146696 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f5bf00517f0 con 0x7f5bf81d64e0 2023-10-30T20:23:09.469 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.468+0000 7f5bedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:09.469 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.468+0000 7f5bedffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:09.539 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.538+0000 7f5bff461640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d0d32a9a-e24a-4d0c-9b90-125485d70672", "id": 1}] 2023-10-30T20:23:09.539 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.538+0000 7f5bff461640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d0d32a9a-e24a-4d0c-9b90-125485d70672", "id": 1}] 2023-10-30T20:23:09.539 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.538+0000 7f5bff461640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:09.539 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.538+0000 7f5bff461640 1 -- 172.21.15.183:0/2865146696 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d0d32a9a-e24a-4d0c-9b90-125485d70672", "id": 1} v 0) v1 -- 0x7f5bf804e9b0 con 0x7f5bf81d64e0 2023-10-30T20:23:09.622 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.620+0000 7f5bedffb640 1 -- 172.21.15.183:0/2865146696 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d0d32a9a-e24a-4d0c-9b90-125485d70672", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5bf0009b10 con 0x7f5bf81d64e0 2023-10-30T20:23:09.622 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.620+0000 7f5bedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d0d32a9a-e24a-4d0c-9b90-125485d70672", "id": 1}] 2023-10-30T20:23:09.622 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.620+0000 7f5bedffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:09.623 INFO:teuthology.orchestra.run.smithi183.stdout:1 2023-10-30T20:23:09.624 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.621+0000 7f5bff461640 10 librados: watch_flush enter 2023-10-30T20:23:09.624 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.621+0000 7f5bff461640 10 librados: watch_flush exit 2023-10-30T20:23:09.625 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.621+0000 7f5bff461640 20 client.4120.objecter shutdown clearing up homeless session... 2023-10-30T20:23:09.625 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.621+0000 7f5bff461640 10 client.4120.objecter successfully canceled tick 2023-10-30T20:23:09.625 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.621+0000 7f5bff461640 10 monclient: shutdown 2023-10-30T20:23:09.625 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.621+0000 7f5bff461640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:09.625 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.621+0000 7f5bff461640 1 -- 172.21.15.183:0/2865146696 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bf81d64e0 msgr2=0x7f5bf813f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:09.626 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.621+0000 7f5bff461640 1 --2- 172.21.15.183:0/2865146696 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bf81d64e0 0x7f5bf813f820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5bf002c4f0 tx=0x7f5bf0004a40 comp rx=0 tx=0).stop 2023-10-30T20:23:09.626 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.621+0000 7f5bff461640 1 -- 172.21.15.183:0/2865146696 shutdown_connections 2023-10-30T20:23:09.626 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.621+0000 7f5bff461640 1 --2- 172.21.15.183:0/2865146696 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bf81d64e0 0x7f5bf813f820 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.626 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.621+0000 7f5bff461640 1 --2- 172.21.15.183:0/2865146696 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf80acf20 0x7f5bf81d6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.626 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.622+0000 7f5bff461640 1 --2- 172.21.15.183:0/2865146696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf8065ab0 0x7f5bf81cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.626 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.622+0000 7f5bff461640 1 -- 172.21.15.183:0/2865146696 >> 172.21.15.183:0/2865146696 conn(0x7f5bf80b3820 msgr2=0x7f5bf80737b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:09.627 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.622+0000 7f5bff461640 1 -- 172.21.15.183:0/2865146696 shutdown_connections 2023-10-30T20:23:09.627 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.622+0000 7f5bff461640 1 -- 172.21.15.183:0/2865146696 wait complete. 2023-10-30T20:23:09.627 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.622+0000 7f5bff461640 1 librados: shutdown 2023-10-30T20:23:09.634 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 508f2c3a-9e54-46aa-b6c0-52fad97fd341 2 2023-10-30T20:23:09.661 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:09.660+0000 7fea71613040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-30T20:23:09.772 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.770+0000 7ffa5b17c640 10 monclient: get_monmap_and_config 2023-10-30T20:23:09.772 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.770+0000 7ffa5b17c640 10 monclient: build_initial_monmap 2023-10-30T20:23:09.772 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.770+0000 7ffa5b17c640 10 monclient: monmap: 2023-10-30T20:23:09.772 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:09.773 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:09.773 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:23:09.771785+0000 2023-10-30T20:23:09.773 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:23:09.771785+0000 2023-10-30T20:23:09.773 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:09.773 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:09.774 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:09.774 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:09.774 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:09.774 INFO:teuthology.orchestra.run.smithi183.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:09.774 INFO:teuthology.orchestra.run.smithi183.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:09.775 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:09.777 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.771+0000 7ffa5b17c640 1 Processor -- start 2023-10-30T20:23:09.778 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.771+0000 7ffa5b17c640 1 -- start start 2023-10-30T20:23:09.778 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.771+0000 7ffa5b17c640 10 monclient: init 2023-10-30T20:23:09.778 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:09.778 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-10-30T20:23:09.778 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ffa540b4440 0x7ffa54144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:09.779 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 10 monclient(hunting): picked mon.noname-c con 0x7ffa540b4440 addr v2:172.21.15.183:3300/0 2023-10-30T20:23:09.779 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ffa541453e0 0x7ffa540b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:09.779 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 10 monclient(hunting): picked mon.noname-a con 0x7ffa541453e0 addr v2:172.21.15.69:3300/0 2023-10-30T20:23:09.779 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa540b2060 0x7ffa540b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:09.779 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 10 monclient(hunting): picked mon.noname-b con 0x7ffa540b2060 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:09.780 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:09.780 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa54067a10 con 0x7ffa541453e0 2023-10-30T20:23:09.780 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:09.780 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa540b29e0 con 0x7ffa540b4440 2023-10-30T20:23:09.781 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:09.781 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa540a9da0 con 0x7ffa540b2060 2023-10-30T20:23:09.781 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:09.781 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa5b17c640 10 monclient(hunting): authenticate will time out at 1508.974335s 2023-10-30T20:23:09.781 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa58f18640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ffa540b4440 0x7ffa54144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:09.782 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.772+0000 7ffa59719640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa540b2060 0x7ffa540b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:09.782 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.773+0000 7ffa59719640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa540b2060 0x7ffa540b24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.183:51190/0 (socket says 172.21.15.183:51190) 2023-10-30T20:23:09.782 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.773+0000 7ffa59719640 1 -- 172.21.15.183:0/2063542029 learned_addr learned my addr 172.21.15.183:0/2063542029 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-10-30T20:23:09.782 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.773+0000 7ffa59719640 10 monclient(hunting): get_auth_request con 0x7ffa540b2060 auth_method 0 2023-10-30T20:23:09.782 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.773+0000 7ffa59719640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:09.783 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.773+0000 7ffa59719640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:09.783 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.773+0000 7ffa59719640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:09.783 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.773+0000 7ffa58f18640 10 monclient(hunting): get_auth_request con 0x7ffa540b4440 auth_method 0 2023-10-30T20:23:09.783 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.773+0000 7ffa58f18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:09.783 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.773+0000 7ffa58f18640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:09.784 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.773+0000 7ffa58f18640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:09.784 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.773+0000 7ffa53fff640 1 --2- 172.21.15.183:0/2063542029 >> v2:172.21.15.69:3300/0 conn(0x7ffa541453e0 0x7ffa540b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:09.784 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.773+0000 7ffa59719640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:09.784 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.773+0000 7ffa59719640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:09.784 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.773+0000 7ffa59719640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:09.785 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.774+0000 7ffa53fff640 10 monclient(hunting): get_auth_request con 0x7ffa541453e0 auth_method 0 2023-10-30T20:23:09.785 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.774+0000 7ffa53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:09.785 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.774+0000 7ffa53fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:09.785 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.774+0000 7ffa53fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:09.785 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.774+0000 7ffa53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:09.786 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.774+0000 7ffa53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:09.786 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.774+0000 7ffa53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:09.786 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.775+0000 7ffa58f18640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:09.786 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.775+0000 7ffa58f18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:09.786 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.775+0000 7ffa58f18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:09.787 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.775+0000 7ffa59719640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2023-10-30T20:23:09.787 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.775+0000 7ffa59719640 1 -- 172.21.15.183:0/2063542029 >> v2:172.21.15.183:3300/0 conn(0x7ffa540b4440 msgr2=0x7ffa54144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:09.787 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.775+0000 7ffa59719640 1 --2- 172.21.15.183:0/2063542029 >> v2:172.21.15.183:3300/0 conn(0x7ffa540b4440 0x7ffa54144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.787 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.775+0000 7ffa59719640 1 -- 172.21.15.183:0/2063542029 >> v2:172.21.15.69:3300/0 conn(0x7ffa541453e0 msgr2=0x7ffa540b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:09.788 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.775+0000 7ffa59719640 1 --2- 172.21.15.183:0/2063542029 >> v2:172.21.15.69:3300/0 conn(0x7ffa541453e0 0x7ffa540b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.788 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.775+0000 7ffa59719640 10 monclient: _finish_hunting 0 2023-10-30T20:23:09.788 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.775+0000 7ffa59719640 1 monclient: found mon.noname-b 2023-10-30T20:23:09.788 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.775+0000 7ffa59719640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:09.788 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.775+0000 7ffa59719640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:23:09.789 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.775+0000 7ffa59719640 1 -- 172.21.15.183:0/2063542029 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa540aa080 con 0x7ffa540b2060 2023-10-30T20:23:09.793 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.777+0000 7ffa59719640 1 --2- 172.21.15.183:0/2063542029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa540b2060 0x7ffa540b24a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ffa54066a50 tx=0x7ffa4c02bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcbefd728aea4d84 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:09.793 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa537fe640 1 -- 172.21.15.183:0/2063542029 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c02c950 con 0x7ffa540b2060 2023-10-30T20:23:09.793 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:09.793 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:23:09.793 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa537fe640 10 monclient: dump: 2023-10-30T20:23:09.794 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:09.794 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:09.794 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.794 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.794 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:09.795 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:09.795 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:09.795 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:09.795 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:09.795 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:09.795 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa537fe640 10 monclient: _finish_auth 0 2023-10-30T20:23:09.796 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa537fe640 10 monclient: _check_auth_tickets 2023-10-30T20:23:09.796 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:09.796 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa537fe640 1 -- 172.21.15.183:0/2063542029 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4c02cac0 con 0x7ffa540b2060 2023-10-30T20:23:09.796 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa537fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:09.796 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa537fe640 1 -- 172.21.15.183:0/2063542029 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c02ce60 con 0x7ffa540b2060 2023-10-30T20:23:09.797 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa5b17c640 5 monclient: authenticate success, global_id 4112 2023-10-30T20:23:09.797 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa5b17c640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:09.797 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa5b17c640 4 set_mon_vals no callback set 2023-10-30T20:23:09.797 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:09.797 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:23:09.798 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa537fe640 10 monclient: dump: 2023-10-30T20:23:09.798 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:09.798 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:09.798 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.798 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.798 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:09.799 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:09.799 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:09.799 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:09.799 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:09.799 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:09.800 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa5b17c640 10 monclient: shutdown 2023-10-30T20:23:09.800 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa5b17c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:09.800 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa5b17c640 1 -- 172.21.15.183:0/2063542029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa540b2060 msgr2=0x7ffa540b24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:09.800 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.781+0000 7ffa5b17c640 1 --2- 172.21.15.183:0/2063542029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa540b2060 0x7ffa540b24a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ffa54066a50 tx=0x7ffa4c02bdb0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.800 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.782+0000 7ffa5b17c640 1 -- 172.21.15.183:0/2063542029 shutdown_connections 2023-10-30T20:23:09.801 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.782+0000 7ffa5b17c640 1 --2- 172.21.15.183:0/2063542029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa540b2060 0x7ffa540b24a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.801 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.782+0000 7ffa5b17c640 1 --2- 172.21.15.183:0/2063542029 >> v2:172.21.15.69:3300/0 conn(0x7ffa541453e0 0x7ffa540b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.801 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.782+0000 7ffa5b17c640 1 --2- 172.21.15.183:0/2063542029 >> v2:172.21.15.183:3300/0 conn(0x7ffa540b4440 0x7ffa54144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.801 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.782+0000 7ffa5b17c640 1 -- 172.21.15.183:0/2063542029 >> 172.21.15.183:0/2063542029 conn(0x7ffa54065ea0 msgr2=0x7ffa540662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:09.801 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.782+0000 7ffa5b17c640 1 -- 172.21.15.183:0/2063542029 shutdown_connections 2023-10-30T20:23:09.802 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.782+0000 7ffa5b17c640 1 -- 172.21.15.183:0/2063542029 wait complete. 2023-10-30T20:23:09.802 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.782+0000 7ffa5b17c640 10 monclient: build_initial_monmap 2023-10-30T20:23:09.802 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.782+0000 7ffa5b17c640 10 monclient: monmap: 2023-10-30T20:23:09.802 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:09.803 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:09.803 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2023-10-30T20:23:09.803 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2023-10-30T20:23:09.803 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:09.803 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2023-10-30T20:23:09.804 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:09.804 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:09.804 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:09.804 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:09.804 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 1 librados: starting msgr at 2023-10-30T20:23:09.805 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 1 librados: starting objecter 2023-10-30T20:23:09.805 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 1 Processor -- start 2023-10-30T20:23:09.805 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 1 -- start start 2023-10-30T20:23:09.805 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 1 librados: setting wanted keys 2023-10-30T20:23:09.805 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 1 librados: calling monclient init 2023-10-30T20:23:09.806 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 10 monclient: init 2023-10-30T20:23:09.806 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:09.806 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:23:09.806 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa540b2060 0x7ffa540b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:09.806 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 10 monclient(hunting): picked mon.noname-a con 0x7ffa540b2060 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:09.807 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa540b4440 0x7ffa541ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:09.807 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 10 monclient(hunting): picked mon.noname-b con 0x7ffa540b4440 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:09.807 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa541453e0 0x7ffa5413b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:09.807 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 10 monclient(hunting): picked mon.noname-c con 0x7ffa541453e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:09.807 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:09.807 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa540b1cb0 con 0x7ffa540b2060 2023-10-30T20:23:09.808 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:09.808 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa541cd5c0 con 0x7ffa540b4440 2023-10-30T20:23:09.808 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:09.808 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa541cb890 con 0x7ffa541453e0 2023-10-30T20:23:09.808 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:09.808 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.783+0000 7ffa5b17c640 10 monclient(hunting): authenticate will time out at 1508.985885s 2023-10-30T20:23:09.809 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa58f18640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa540b2060 0x7ffa540b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:09.809 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa53fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa540b4440 0x7ffa541ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:09.809 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa59719640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa541453e0 0x7ffa5413b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:09.809 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa58f18640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa540b2060 0x7ffa540b0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.183:55962/0 (socket says 172.21.15.183:55962) 2023-10-30T20:23:09.809 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa58f18640 1 -- 172.21.15.183:0/2545169688 learned_addr learned my addr 172.21.15.183:0/2545169688 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-10-30T20:23:09.810 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa58f18640 10 monclient(hunting): get_auth_request con 0x7ffa540b2060 auth_method 0 2023-10-30T20:23:09.810 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa58f18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:09.810 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa58f18640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:09.810 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa58f18640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:09.810 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa53fff640 10 monclient(hunting): get_auth_request con 0x7ffa540b4440 auth_method 0 2023-10-30T20:23:09.810 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:09.811 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa53fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:09.811 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa53fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:09.811 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa59719640 10 monclient(hunting): get_auth_request con 0x7ffa541453e0 auth_method 0 2023-10-30T20:23:09.811 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa59719640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:09.811 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa59719640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:09.812 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.784+0000 7ffa59719640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:09.812 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.785+0000 7ffa58f18640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:09.812 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.785+0000 7ffa58f18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:09.812 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.785+0000 7ffa58f18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:09.812 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.785+0000 7ffa59719640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:09.813 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.785+0000 7ffa59719640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:09.813 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.785+0000 7ffa59719640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:09.813 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.785+0000 7ffa53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:09.813 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.785+0000 7ffa53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:09.813 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.785+0000 7ffa53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:09.814 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.787+0000 7ffa58f18640 10 monclient(hunting): handle_auth_done global_id 4119 payload 824 2023-10-30T20:23:09.814 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.787+0000 7ffa58f18640 1 -- 172.21.15.183:0/2545169688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa541453e0 msgr2=0x7ffa5413b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:09.814 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.787+0000 7ffa58f18640 1 --2- 172.21.15.183:0/2545169688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa541453e0 0x7ffa5413b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.814 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.787+0000 7ffa58f18640 1 -- 172.21.15.183:0/2545169688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa540b4440 msgr2=0x7ffa541ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:09.814 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.787+0000 7ffa58f18640 1 --2- 172.21.15.183:0/2545169688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa540b4440 0x7ffa541ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:09.815 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.787+0000 7ffa58f18640 10 monclient: _finish_hunting 0 2023-10-30T20:23:09.815 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.787+0000 7ffa58f18640 1 monclient: found mon.noname-a 2023-10-30T20:23:09.815 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.788+0000 7ffa58f18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:09.815 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.788+0000 7ffa58f18640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:23:09.815 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.788+0000 7ffa58f18640 1 -- 172.21.15.183:0/2545169688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa541cd160 con 0x7ffa540b2060 2023-10-30T20:23:09.816 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:09.796+0000 7fea71613040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T20:23:09.816 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.794+0000 7ffa58f18640 1 --2- 172.21.15.183:0/2545169688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa540b2060 0x7ffa540b0c80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800e9b0 tx=0x7ffa4800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:09.816 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.794+0000 7ffa517fa640 10 client.?.objecter ms_handle_connect 0x7ffa540b2060 2023-10-30T20:23:09.817 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.794+0000 7ffa517fa640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:09.817 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:09.802+0000 7f594b411040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-30T20:23:09.818 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa517fa640 1 -- 172.21.15.183:0/2545169688 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4800c9a0 con 0x7ffa540b2060 2023-10-30T20:23:09.819 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:09.819 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa517fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:23:09.819 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa517fa640 10 monclient: dump: 2023-10-30T20:23:09.819 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:09.819 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:09.820 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.820 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.820 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:09.820 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:09.820 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:09.821 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:09.821 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:09.821 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:09.821 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa517fa640 10 monclient: _finish_auth 0 2023-10-30T20:23:09.821 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa517fa640 10 monclient: _check_auth_tickets 2023-10-30T20:23:09.822 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:09.822 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa517fa640 1 -- 172.21.15.183:0/2545169688 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4800d040 con 0x7ffa540b2060 2023-10-30T20:23:09.822 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa5b17c640 5 monclient: authenticate success, global_id 4119 2023-10-30T20:23:09.822 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa5b17c640 10 monclient: _renew_subs 2023-10-30T20:23:09.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa5b17c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:09.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa5b17c640 1 -- 172.21.15.183:0/2545169688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa5413b830 con 0x7ffa540b2060 2023-10-30T20:23:09.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa517fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:09.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa517fa640 1 -- 172.21.15.183:0/2545169688 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4800cd50 con 0x7ffa540b2060 2023-10-30T20:23:09.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:09.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:23:09.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa517fa640 10 monclient: dump: 2023-10-30T20:23:09.824 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:09.824 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:09.824 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.824 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:09.825 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:09.825 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:09.825 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:09.825 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:09.825 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:09.825 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:09.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa52ffd640 4 set_mon_vals no callback set 2023-10-30T20:23:09.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa5b17c640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:09.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa5b17c640 10 monclient: _renew_subs 2023-10-30T20:23:09.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa5b17c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:09.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa5b17c640 1 -- 172.21.15.183:0/2545169688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa5413bd50 con 0x7ffa540b2060 2023-10-30T20:23:09.827 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.803+0000 7ffa5b17c640 1 librados: init done 2023-10-30T20:23:09.827 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.804+0000 7ffa5b17c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:09.827 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.804+0000 7ffa5b17c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:09.827 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.804+0000 7ffa5b17c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:09.827 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.804+0000 7ffa5b17c640 1 -- 172.21.15.183:0/2545169688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa5404e9b0 con 0x7ffa540b2060 2023-10-30T20:23:09.828 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.819+0000 7ffa517fa640 1 -- 172.21.15.183:0/2545169688 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ffa48005c90 con 0x7ffa540b2060 2023-10-30T20:23:09.828 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.819+0000 7ffa517fa640 1 -- 172.21.15.183:0/2545169688 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7ffa480067f0 con 0x7ffa540b2060 2023-10-30T20:23:09.828 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.819+0000 7ffa517fa640 10 client.4119.objecter ms_dispatch 0x7ffa54000b90 osd_map(4..4 src has 1..4) v4 2023-10-30T20:23:09.829 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.819+0000 7ffa517fa640 3 client.4119.objecter handle_osd_map got epochs [4,4] > 0 2023-10-30T20:23:09.829 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.819+0000 7ffa517fa640 3 client.4119.objecter handle_osd_map decoding full epoch 4 2023-10-30T20:23:09.829 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.819+0000 7ffa517fa640 20 client.4119.objecter dump_active .. 0 homeless 2023-10-30T20:23:09.917 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:09.916+0000 7fea71613040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T20:23:09.934 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.933+0000 7ffa517fa640 1 -- 172.21.15.183:0/2545169688 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7ffa48006030 con 0x7ffa540b2060 2023-10-30T20:23:09.935 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.933+0000 7ffa517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:09.935 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:09.933+0000 7ffa517fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:09.935 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:09.934+0000 7f594b411040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T20:23:09.963 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:09.961+0000 7fea71613040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T20:23:10.006 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.005+0000 7ffa5b17c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "508f2c3a-9e54-46aa-b6c0-52fad97fd341", "id": 2}] 2023-10-30T20:23:10.006 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.005+0000 7ffa5b17c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "508f2c3a-9e54-46aa-b6c0-52fad97fd341", "id": 2}] 2023-10-30T20:23:10.006 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.005+0000 7ffa5b17c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:10.006 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.005+0000 7ffa5b17c640 1 -- 172.21.15.183:0/2545169688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "508f2c3a-9e54-46aa-b6c0-52fad97fd341", "id": 2} v 0) v1 -- 0x7ffa541cd360 con 0x7ffa540b2060 2023-10-30T20:23:10.053 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.052+0000 7ffa517fa640 1 -- 172.21.15.183:0/2545169688 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "508f2c3a-9e54-46aa-b6c0-52fad97fd341", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7ffa48006260 con 0x7ffa540b2060 2023-10-30T20:23:10.053 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.052+0000 7ffa517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "508f2c3a-9e54-46aa-b6c0-52fad97fd341", "id": 2}] 2023-10-30T20:23:10.054 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.052+0000 7ffa517fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:10.054 INFO:teuthology.orchestra.run.smithi183.stdout:2 2023-10-30T20:23:10.055 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 10 librados: watch_flush enter 2023-10-30T20:23:10.056 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 10 librados: watch_flush exit 2023-10-30T20:23:10.056 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 20 client.4119.objecter shutdown clearing up homeless session... 2023-10-30T20:23:10.056 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 10 client.4119.objecter successfully canceled tick 2023-10-30T20:23:10.056 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 10 monclient: shutdown 2023-10-30T20:23:10.057 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:10.057 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 1 -- 172.21.15.183:0/2545169688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa540b2060 msgr2=0x7ffa540b0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:10.057 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 1 --2- 172.21.15.183:0/2545169688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa540b2060 0x7ffa540b0c80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800e9b0 tx=0x7ffa4800ee70 comp rx=0 tx=0).stop 2023-10-30T20:23:10.057 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 1 -- 172.21.15.183:0/2545169688 shutdown_connections 2023-10-30T20:23:10.057 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 1 --2- 172.21.15.183:0/2545169688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa541453e0 0x7ffa5413b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.057 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 1 --2- 172.21.15.183:0/2545169688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa540b4440 0x7ffa541ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.058 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 1 --2- 172.21.15.183:0/2545169688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa540b2060 0x7ffa540b0c80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.058 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 1 -- 172.21.15.183:0/2545169688 >> 172.21.15.183:0/2545169688 conn(0x7ffa54065ea0 msgr2=0x7ffa540b78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:10.058 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 1 -- 172.21.15.183:0/2545169688 shutdown_connections 2023-10-30T20:23:10.058 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 1 -- 172.21.15.183:0/2545169688 wait complete. 2023-10-30T20:23:10.059 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.053+0000 7ffa5b17c640 1 librados: shutdown 2023-10-30T20:23:10.064 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:10.063+0000 7fea71613040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T20:23:10.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 003a2613-1050-4a96-b927-0e5983687319 3 2023-10-30T20:23:10.069 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:10.068+0000 7f594b411040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T20:23:10.117 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:10.116+0000 7f594b411040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T20:23:10.158 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:10.157+0000 7fea71613040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T20:23:10.167 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.166+0000 7faa8fc56640 10 monclient: get_monmap_and_config 2023-10-30T20:23:10.167 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.166+0000 7faa8fc56640 10 monclient: build_initial_monmap 2023-10-30T20:23:10.168 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.166+0000 7faa8fc56640 10 monclient: monmap: 2023-10-30T20:23:10.168 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:10.168 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:10.168 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:23:10.167090+0000 2023-10-30T20:23:10.168 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:23:10.167090+0000 2023-10-30T20:23:10.169 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:10.169 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:10.169 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:10.169 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:10.169 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:10.170 INFO:teuthology.orchestra.run.smithi183.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:10.170 INFO:teuthology.orchestra.run.smithi183.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:10.170 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:10.170 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.166+0000 7faa8fc56640 1 Processor -- start 2023-10-30T20:23:10.172 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.166+0000 7faa8fc56640 1 -- start start 2023-10-30T20:23:10.173 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.166+0000 7faa8fc56640 10 monclient: init 2023-10-30T20:23:10.173 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.167+0000 7faa8fc56640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:10.173 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.167+0000 7faa8fc56640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-10-30T20:23:10.173 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.167+0000 7faa8fc56640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7faa880b4440 0x7faa88144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:10.173 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.167+0000 7faa8fc56640 10 monclient(hunting): picked mon.noname-c con 0x7faa880b4440 addr v2:172.21.15.183:3300/0 2023-10-30T20:23:10.174 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.167+0000 7faa8fc56640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa881453e0 0x7faa880b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:10.174 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.167+0000 7faa8fc56640 10 monclient(hunting): picked mon.noname-b con 0x7faa881453e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:10.174 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.167+0000 7faa8fc56640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa880b2060 addr v1:172.21.15.69:6789/0 2023-10-30T20:23:10.174 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.167+0000 7faa8fc56640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa880a9da0 con 0x7faa880b2060 2023-10-30T20:23:10.174 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.167+0000 7faa8fc56640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:10.175 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.167+0000 7faa8fc56640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa88067a10 con 0x7faa880b4440 2023-10-30T20:23:10.175 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.167+0000 7faa8fc56640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:10.175 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.167+0000 7faa8fc56640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa880aa080 con 0x7faa881453e0 2023-10-30T20:23:10.175 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.167+0000 7faa8fc56640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:10.175 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.167+0000 7faa8fc56640 10 monclient(hunting): authenticate will time out at 1509.369615s 2023-10-30T20:23:10.176 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.168+0000 7faa8d9f2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7faa880b4440 0x7faa88144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:10.176 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.168+0000 7faa8d1f1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa881453e0 0x7faa880b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:10.176 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.168+0000 7faa8e1f3640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7faa880b2060 0x7faa8813e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.183:36800/0 (socket says 172.21.15.183:36800) 2023-10-30T20:23:10.176 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.168+0000 7faa8e1f3640 1 -- 172.21.15.183:0/1551137606 learned_addr learned my addr 172.21.15.183:0/1551137606 (peer_addr_for_me v1:172.21.15.183:0/0) 2023-10-30T20:23:10.176 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.168+0000 7faa8d1f1640 10 monclient(hunting): get_auth_request con 0x7faa881453e0 auth_method 0 2023-10-30T20:23:10.177 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.168+0000 7faa8d1f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:10.177 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.168+0000 7faa8d1f1640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:10.177 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.168+0000 7faa8d1f1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:10.177 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.168+0000 7faa8d9f2640 10 monclient(hunting): get_auth_request con 0x7faa880b4440 auth_method 0 2023-10-30T20:23:10.178 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.168+0000 7faa8d9f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:10.178 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.168+0000 7faa8d9f2640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:10.178 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.168+0000 7faa8d9f2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:10.178 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.169+0000 7faa8d1f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:10.178 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.169+0000 7faa8d1f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:10.179 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.169+0000 7faa8d1f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:10.179 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.169+0000 7faa8d9f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:10.179 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.169+0000 7faa8d9f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:10.179 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.169+0000 7faa8d9f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:10.179 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.170+0000 7faa8d9f2640 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2023-10-30T20:23:10.180 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.170+0000 7faa8d9f2640 1 -- 172.21.15.183:0/1551137606 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa881453e0 msgr2=0x7faa880b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:10.180 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.170+0000 7faa8d9f2640 1 --2- 172.21.15.183:0/1551137606 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa881453e0 0x7faa880b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.180 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.170+0000 7faa8d9f2640 1 -- 172.21.15.183:0/1551137606 >> v1:172.21.15.69:6789/0 conn(0x7faa880b2060 legacy=0x7faa8813e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:10.180 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.170+0000 7faa8d9f2640 10 monclient: _finish_hunting 0 2023-10-30T20:23:10.181 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.170+0000 7faa8d9f2640 1 monclient: found mon.noname-c 2023-10-30T20:23:10.181 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.170+0000 7faa8d9f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:10.181 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.170+0000 7faa8d9f2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:23:10.181 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.170+0000 7faa8d9f2640 1 -- 172.21.15.183:0/1551137606 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa880aa360 con 0x7faa880b4440 2023-10-30T20:23:10.185 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.172+0000 7faa8d9f2640 1 --2- 172.21.15.183:0/1551137606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa880b4440 0x7faa88144ea0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faa840097d0 tx=0x7faa8402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c67d38c2f4418d75 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:10.185 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8c9f0640 1 -- 172.21.15.183:0/1551137606 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa840316f0 con 0x7faa880b4440 2023-10-30T20:23:10.185 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8c9f0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:10.185 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8c9f0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:23:10.186 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8c9f0640 10 monclient: dump: 2023-10-30T20:23:10.186 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:10.186 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:10.186 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.186 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.187 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:10.187 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:10.187 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:10.187 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:10.187 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:10.187 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:10.188 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8c9f0640 10 monclient: _finish_auth 0 2023-10-30T20:23:10.188 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8c9f0640 10 monclient: _check_auth_tickets 2023-10-30T20:23:10.188 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8c9f0640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:10.188 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8c9f0640 1 -- 172.21.15.183:0/1551137606 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa84031860 con 0x7faa880b4440 2023-10-30T20:23:10.188 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8c9f0640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:10.188 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8c9f0640 1 -- 172.21.15.183:0/1551137606 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84031c00 con 0x7faa880b4440 2023-10-30T20:23:10.189 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8fc56640 5 monclient: authenticate success, global_id 4129 2023-10-30T20:23:10.189 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8fc56640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:10.189 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8fc56640 4 set_mon_vals no callback set 2023-10-30T20:23:10.189 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8c9f0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:10.189 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8c9f0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:23:10.190 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8c9f0640 10 monclient: dump: 2023-10-30T20:23:10.190 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:10.190 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:10.190 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.190 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.191 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:10.191 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:10.191 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:10.191 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:10.191 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:10.192 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:10.192 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8fc56640 10 monclient: shutdown 2023-10-30T20:23:10.192 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8fc56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:10.192 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8fc56640 1 -- 172.21.15.183:0/1551137606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa880b4440 msgr2=0x7faa88144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:10.192 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8fc56640 1 --2- 172.21.15.183:0/1551137606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa880b4440 0x7faa88144ea0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faa840097d0 tx=0x7faa8402b790 comp rx=0 tx=0).stop 2023-10-30T20:23:10.193 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8fc56640 1 -- 172.21.15.183:0/1551137606 shutdown_connections 2023-10-30T20:23:10.193 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8fc56640 1 --2- 172.21.15.183:0/1551137606 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa881453e0 0x7faa880b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.193 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8fc56640 1 --2- 172.21.15.183:0/1551137606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa880b4440 0x7faa88144ea0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.193 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8fc56640 1 -- 172.21.15.183:0/1551137606 >> 172.21.15.183:0/1551137606 conn(0x7faa88065ea0 msgr2=0x7faa880662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:10.193 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8fc56640 1 -- 172.21.15.183:0/1551137606 shutdown_connections 2023-10-30T20:23:10.194 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.176+0000 7faa8fc56640 1 -- 172.21.15.183:0/1551137606 wait complete. 2023-10-30T20:23:10.194 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.177+0000 7faa8fc56640 10 monclient: build_initial_monmap 2023-10-30T20:23:10.194 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.177+0000 7faa8fc56640 10 monclient: monmap: 2023-10-30T20:23:10.194 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:10.194 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:10.195 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2023-10-30T20:23:10.195 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2023-10-30T20:23:10.195 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:10.195 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2023-10-30T20:23:10.195 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:10.195 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:10.196 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:10.196 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:10.196 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.177+0000 7faa8fc56640 1 librados: starting msgr at 2023-10-30T20:23:10.196 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.177+0000 7faa8fc56640 1 librados: starting objecter 2023-10-30T20:23:10.196 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.177+0000 7faa8fc56640 1 Processor -- start 2023-10-30T20:23:10.197 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.177+0000 7faa8fc56640 1 -- start start 2023-10-30T20:23:10.197 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.177+0000 7faa8fc56640 1 librados: setting wanted keys 2023-10-30T20:23:10.197 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.177+0000 7faa8fc56640 1 librados: calling monclient init 2023-10-30T20:23:10.197 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.177+0000 7faa8fc56640 10 monclient: init 2023-10-30T20:23:10.197 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:10.197 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T20:23:10.198 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa880b2060 0x7faa881ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:10.198 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 10 monclient(hunting): picked mon.noname-b con 0x7faa880b2060 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:10.198 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa880b4440 0x7faa881cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:10.198 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 10 monclient(hunting): picked mon.noname-a con 0x7faa880b4440 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:10.198 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa881453e0 0x7faa8813a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:10.199 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 10 monclient(hunting): picked mon.noname-c con 0x7faa881453e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:10.199 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:10.199 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa880a8b50 con 0x7faa880b4440 2023-10-30T20:23:10.199 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:10.199 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa881cdf50 con 0x7faa880b2060 2023-10-30T20:23:10.200 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:10.200 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8813ffa0 con 0x7faa881453e0 2023-10-30T20:23:10.200 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:10.200 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.178+0000 7faa8fc56640 10 monclient(hunting): authenticate will time out at 1509.380576s 2023-10-30T20:23:10.200 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8e1f3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa881453e0 0x7faa8813a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:10.201 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8d1f1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa880b4440 0x7faa881cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:10.201 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8d9f2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa880b2060 0x7faa881ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:10.201 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8e1f3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa881453e0 0x7faa8813a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.183:51222/0 (socket says 172.21.15.183:51222) 2023-10-30T20:23:10.201 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8e1f3640 1 -- 172.21.15.183:0/464678255 learned_addr learned my addr 172.21.15.183:0/464678255 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-10-30T20:23:10.202 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8e1f3640 10 monclient(hunting): get_auth_request con 0x7faa881453e0 auth_method 0 2023-10-30T20:23:10.202 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8e1f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:10.202 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8d1f1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa880b4440 0x7faa881cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.183:55966/0 (socket says 172.21.15.183:55966) 2023-10-30T20:23:10.202 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8e1f3640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:10.203 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8e1f3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:10.203 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8d1f1640 10 monclient(hunting): get_auth_request con 0x7faa880b4440 auth_method 0 2023-10-30T20:23:10.203 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8d1f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:10.203 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8d1f1640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:10.203 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8d1f1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:10.204 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8d9f2640 10 monclient(hunting): get_auth_request con 0x7faa880b2060 auth_method 0 2023-10-30T20:23:10.204 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8d9f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:10.204 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8d9f2640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:10.204 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8d9f2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:10.204 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8e1f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:10.204 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8e1f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:10.205 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.179+0000 7faa8e1f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:10.205 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.180+0000 7faa8d1f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:10.205 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.180+0000 7faa8d1f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:10.205 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.180+0000 7faa8d1f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:10.205 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.180+0000 7faa8d9f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:10.206 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.180+0000 7faa8d9f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:10.206 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.180+0000 7faa8d9f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:10.206 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.182+0000 7faa8e1f3640 10 monclient(hunting): handle_auth_done global_id 4121 payload 824 2023-10-30T20:23:10.206 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.182+0000 7faa8e1f3640 1 -- 172.21.15.183:0/464678255 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa880b2060 msgr2=0x7faa881ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:10.206 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.182+0000 7faa8e1f3640 1 --2- 172.21.15.183:0/464678255 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa880b2060 0x7faa881ccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.207 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.182+0000 7faa8e1f3640 1 -- 172.21.15.183:0/464678255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa880b4440 msgr2=0x7faa881cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:10.207 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.182+0000 7faa8e1f3640 1 --2- 172.21.15.183:0/464678255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa880b4440 0x7faa881cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.207 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.182+0000 7faa8e1f3640 10 monclient: _finish_hunting 0 2023-10-30T20:23:10.207 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.182+0000 7faa8e1f3640 1 monclient: found mon.noname-c 2023-10-30T20:23:10.207 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.182+0000 7faa8e1f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:10.208 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.182+0000 7faa8e1f3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:23:10.208 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.182+0000 7faa8e1f3640 1 -- 172.21.15.183:0/464678255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa88140220 con 0x7faa881453e0 2023-10-30T20:23:10.209 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.184+0000 7faa8e1f3640 1 --2- 172.21.15.183:0/464678255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa881453e0 0x7faa8813a7a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faa8000cbc0 tx=0x7faa8000a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:10.210 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.184+0000 7faa7e7fc640 10 client.?.objecter ms_handle_connect 0x7faa881453e0 2023-10-30T20:23:10.210 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.185+0000 7faa7e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:10.210 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa7e7fc640 1 -- 172.21.15.183:0/464678255 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8000ad40 con 0x7faa881453e0 2023-10-30T20:23:10.210 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:10.210 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa7e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:23:10.211 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa7e7fc640 10 monclient: dump: 2023-10-30T20:23:10.211 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:10.211 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:10.211 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.211 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.211 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:10.212 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:10.212 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:10.212 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:10.212 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:10.212 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:10.213 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa7e7fc640 10 monclient: _finish_auth 0 2023-10-30T20:23:10.213 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa7e7fc640 10 monclient: _check_auth_tickets 2023-10-30T20:23:10.213 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:10.213 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa7e7fc640 1 -- 172.21.15.183:0/464678255 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8000e040 con 0x7faa881453e0 2023-10-30T20:23:10.213 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa7e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:10.213 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa7e7fc640 1 -- 172.21.15.183:0/464678255 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa80012600 con 0x7faa881453e0 2023-10-30T20:23:10.214 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa8fc56640 5 monclient: authenticate success, global_id 4121 2023-10-30T20:23:10.214 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa8fc56640 10 monclient: _renew_subs 2023-10-30T20:23:10.214 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa8fc56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:10.214 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa8fc56640 1 -- 172.21.15.183:0/464678255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa881ce400 con 0x7faa881453e0 2023-10-30T20:23:10.214 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa7ffff640 4 set_mon_vals no callback set 2023-10-30T20:23:10.214 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:10.215 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:23:10.215 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa7e7fc640 10 monclient: dump: 2023-10-30T20:23:10.215 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:10.215 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:10.215 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.216 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.216 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:10.216 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:10.216 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:10.216 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:10.216 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:10.217 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:10.217 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa8fc56640 10 client.4121.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:10.217 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa8fc56640 10 monclient: _renew_subs 2023-10-30T20:23:10.217 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa8fc56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:10.217 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa8fc56640 1 -- 172.21.15.183:0/464678255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8813aec0 con 0x7faa881453e0 2023-10-30T20:23:10.218 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.188+0000 7faa8fc56640 1 librados: init done 2023-10-30T20:23:10.218 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.189+0000 7faa8fc56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:10.218 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.189+0000 7faa8fc56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:10.218 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.189+0000 7faa8fc56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:10.218 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.189+0000 7faa8fc56640 1 -- 172.21.15.183:0/464678255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8804e9b0 con 0x7faa881453e0 2023-10-30T20:23:10.219 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.192+0000 7faa7e7fc640 1 -- 172.21.15.183:0/464678255 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7faa80012a70 con 0x7faa881453e0 2023-10-30T20:23:10.219 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.192+0000 7faa7e7fc640 1 -- 172.21.15.183:0/464678255 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7faa80007500 con 0x7faa881453e0 2023-10-30T20:23:10.219 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.192+0000 7faa7e7fc640 10 client.4121.objecter ms_dispatch 0x7faa88000b90 osd_map(5..5 src has 1..5) v4 2023-10-30T20:23:10.219 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.192+0000 7faa7e7fc640 3 client.4121.objecter handle_osd_map got epochs [5,5] > 0 2023-10-30T20:23:10.219 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.192+0000 7faa7e7fc640 3 client.4121.objecter handle_osd_map decoding full epoch 5 2023-10-30T20:23:10.220 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.192+0000 7faa7e7fc640 20 client.4121.objecter dump_active .. 0 homeless 2023-10-30T20:23:10.220 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:10.219+0000 7f594b411040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T20:23:10.302 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:10.300+0000 7fea71613040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T20:23:10.314 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:10.313+0000 7f594b411040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T20:23:10.345 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:10.344+0000 7fea71613040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T20:23:10.362 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.360+0000 7faa7e7fc640 1 -- 172.21.15.183:0/464678255 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7faa80012d20 con 0x7faa881453e0 2023-10-30T20:23:10.362 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.360+0000 7faa7e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:10.362 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.360+0000 7faa7e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:10.434 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.432+0000 7faa8fc56640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "003a2613-1050-4a96-b927-0e5983687319", "id": 3}] 2023-10-30T20:23:10.434 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.432+0000 7faa8fc56640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "003a2613-1050-4a96-b927-0e5983687319", "id": 3}] 2023-10-30T20:23:10.434 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.432+0000 7faa8fc56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:10.434 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.432+0000 7faa8fc56640 1 -- 172.21.15.183:0/464678255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "003a2613-1050-4a96-b927-0e5983687319", "id": 3} v 0) v1 -- 0x7faa8813d5c0 con 0x7faa881453e0 2023-10-30T20:23:10.459 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:10.458+0000 7f594b411040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T20:23:10.495 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:10.494+0000 7fea71613040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T20:23:10.502 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:10.501+0000 7f594b411040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T20:23:10.540 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.539+0000 7faa7e7fc640 1 -- 172.21.15.183:0/464678255 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "003a2613-1050-4a96-b927-0e5983687319", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7faa8001c510 con 0x7faa881453e0 2023-10-30T20:23:10.541 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.539+0000 7faa7e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "003a2613-1050-4a96-b927-0e5983687319", "id": 3}] 2023-10-30T20:23:10.541 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.539+0000 7faa7e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:10.542 INFO:teuthology.orchestra.run.smithi183.stdout:3 2023-10-30T20:23:10.543 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 10 librados: watch_flush enter 2023-10-30T20:23:10.543 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 10 librados: watch_flush exit 2023-10-30T20:23:10.543 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 20 client.4121.objecter shutdown clearing up homeless session... 2023-10-30T20:23:10.544 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 10 client.4121.objecter successfully canceled tick 2023-10-30T20:23:10.544 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 10 monclient: shutdown 2023-10-30T20:23:10.544 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:10.544 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 1 -- 172.21.15.183:0/464678255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa881453e0 msgr2=0x7faa8813a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:10.544 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 1 --2- 172.21.15.183:0/464678255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa881453e0 0x7faa8813a7a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faa8000cbc0 tx=0x7faa8000a430 comp rx=0 tx=0).stop 2023-10-30T20:23:10.544 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 1 -- 172.21.15.183:0/464678255 shutdown_connections 2023-10-30T20:23:10.545 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 1 --2- 172.21.15.183:0/464678255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa881453e0 0x7faa8813a7a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.545 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 1 --2- 172.21.15.183:0/464678255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa880b4440 0x7faa881cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.545 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 1 --2- 172.21.15.183:0/464678255 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa880b2060 0x7faa881ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.545 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 1 -- 172.21.15.183:0/464678255 >> 172.21.15.183:0/464678255 conn(0x7faa88065ea0 msgr2=0x7faa880b6090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:10.545 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 1 -- 172.21.15.183:0/464678255 shutdown_connections 2023-10-30T20:23:10.546 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 1 -- 172.21.15.183:0/464678255 wait complete. 2023-10-30T20:23:10.546 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.540+0000 7faa8fc56640 1 librados: shutdown 2023-10-30T20:23:10.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 678fc58e-ed6a-4fd0-9825-93297e3af6f6 4 2023-10-30T20:23:10.649 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.648+0000 7f298de6f640 10 monclient: get_monmap_and_config 2023-10-30T20:23:10.649 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.648+0000 7f298de6f640 10 monclient: build_initial_monmap 2023-10-30T20:23:10.650 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.648+0000 7f298de6f640 10 monclient: monmap: 2023-10-30T20:23:10.650 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:10.650 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:10.650 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:23:10.649356+0000 2023-10-30T20:23:10.654 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:23:10.649356+0000 2023-10-30T20:23:10.654 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:10.654 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:10.654 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:10.654 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:10.655 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:10.655 INFO:teuthology.orchestra.run.smithi183.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:10.655 INFO:teuthology.orchestra.run.smithi183.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:10.655 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:10.656 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 1 Processor -- start 2023-10-30T20:23:10.657 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 1 -- start start 2023-10-30T20:23:10.657 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 10 monclient: init 2023-10-30T20:23:10.657 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:10.657 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-10-30T20:23:10.657 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f29880b4440 addr v1:172.21.15.183:6789/0 2023-10-30T20:23:10.657 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29881455b0 addr v1:172.21.15.69:6789/0 2023-10-30T20:23:10.658 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f298813ed50 0x7f298814d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:10.658 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 10 monclient(hunting): picked mon.noname-a con 0x7f298813ed50 addr v2:172.21.15.69:3300/0 2023-10-30T20:23:10.658 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29880a9da0 con 0x7f29881455b0 2023-10-30T20:23:10.658 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29880aa080 con 0x7f29880b4440 2023-10-30T20:23:10.659 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:10.659 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2988067a10 con 0x7f298813ed50 2023-10-30T20:23:10.659 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:10.659 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f298de6f640 10 monclient(hunting): authenticate will time out at 1509.851930s 2023-10-30T20:23:10.659 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:10.649+0000 7f594b411040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T20:23:10.661 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.650+0000 7f29877fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f29880b4440 0x7f2988144ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:53858/0 (socket says 172.21.15.183:53858) 2023-10-30T20:23:10.661 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.650+0000 7f29877fe640 1 -- 172.21.15.183:0/2092898849 learned_addr learned my addr 172.21.15.183:0/2092898849 (peer_addr_for_me v1:172.21.15.183:0/0) 2023-10-30T20:23:10.661 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.650+0000 7f2987fff640 1 --2- 172.21.15.183:0/2092898849 >> v2:172.21.15.69:3300/0 conn(0x7f298813ed50 0x7f298814d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:10.662 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.651+0000 7f2987fff640 10 monclient(hunting): get_auth_request con 0x7f298813ed50 auth_method 0 2023-10-30T20:23:10.662 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.651+0000 7f2987fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:10.662 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.651+0000 7f2987fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:10.662 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.651+0000 7f2987fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:10.662 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.651+0000 7f2987fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:10.663 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.651+0000 7f2987fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:10.663 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.651+0000 7f2987fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:10.663 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.652+0000 7f29867fc640 1 -- 172.21.15.183:0/2092898849 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734730981 0 0) 0x7f2978001110 con 0x7f29880b4440 2023-10-30T20:23:10.663 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.652+0000 7f29867fc640 10 monclient(hunting): _negotiate 2023-10-30T20:23:10.663 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.652+0000 7f29867fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:10.664 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.652+0000 7f29867fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:10.664 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.652+0000 7f29867fc640 10 monclient(hunting): my global_id is 4135 2023-10-30T20:23:10.664 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.652+0000 7f29867fc640 1 -- 172.21.15.183:0/2092898849 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2970003520 con 0x7f29880b4440 2023-10-30T20:23:10.664 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.653+0000 7f2987fff640 10 monclient(hunting): handle_auth_done global_id 4128 payload 274 2023-10-30T20:23:10.664 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.653+0000 7f2987fff640 1 -- 172.21.15.183:0/2092898849 >> v1:172.21.15.183:6789/0 conn(0x7f29880b4440 legacy=0x7f2988144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:10.664 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.653+0000 7f2987fff640 1 -- 172.21.15.183:0/2092898849 >> v1:172.21.15.69:6789/0 conn(0x7f29881455b0 legacy=0x7f298813e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:10.665 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.653+0000 7f2987fff640 10 monclient: _finish_hunting 0 2023-10-30T20:23:10.665 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.653+0000 7f2987fff640 1 monclient: found mon.noname-a 2023-10-30T20:23:10.665 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.653+0000 7f2987fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:10.665 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.653+0000 7f2987fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:23:10.665 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.653+0000 7f2987fff640 1 -- 172.21.15.183:0/2092898849 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29880aa360 con 0x7f298813ed50 2023-10-30T20:23:10.666 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.658+0000 7f2987fff640 1 --2- 172.21.15.183:0/2092898849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f298813ed50 0x7f298814d3f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2980009d90 tx=0x7f298002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e724a2f68ffefdfc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:10.670 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f29867fc640 1 -- 172.21.15.183:0/2092898849 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298002cbd0 con 0x7f298813ed50 2023-10-30T20:23:10.671 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f29867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:10.671 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f29867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:23:10.671 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f29867fc640 10 monclient: dump: 2023-10-30T20:23:10.671 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:10.672 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:10.672 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.672 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.672 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:10.672 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:10.673 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:10.673 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:10.673 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:10.673 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:10.673 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f29867fc640 10 monclient: _finish_auth 0 2023-10-30T20:23:10.674 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f29867fc640 10 monclient: _check_auth_tickets 2023-10-30T20:23:10.674 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f29867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:10.674 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f29867fc640 1 -- 172.21.15.183:0/2092898849 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298002cd40 con 0x7f298813ed50 2023-10-30T20:23:10.674 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f29867fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:10.674 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f29867fc640 1 -- 172.21.15.183:0/2092898849 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2980034660 con 0x7f298813ed50 2023-10-30T20:23:10.675 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f298de6f640 5 monclient: authenticate success, global_id 4128 2023-10-30T20:23:10.675 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f298de6f640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:10.675 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f298de6f640 4 set_mon_vals no callback set 2023-10-30T20:23:10.675 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f29867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:10.675 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f29867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:23:10.675 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f29867fc640 10 monclient: dump: 2023-10-30T20:23:10.676 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:10.676 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:10.676 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.676 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.676 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:10.677 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:10.677 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:10.677 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:10.677 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:10.677 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:10.678 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f298de6f640 10 monclient: shutdown 2023-10-30T20:23:10.678 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f298de6f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:10.678 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f298de6f640 1 -- 172.21.15.183:0/2092898849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f298813ed50 msgr2=0x7f298814d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:10.678 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f298de6f640 1 --2- 172.21.15.183:0/2092898849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f298813ed50 0x7f298814d3f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2980009d90 tx=0x7f298002be20 comp rx=0 tx=0).stop 2023-10-30T20:23:10.678 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f298de6f640 1 -- 172.21.15.183:0/2092898849 shutdown_connections 2023-10-30T20:23:10.679 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f298de6f640 1 --2- 172.21.15.183:0/2092898849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f298813ed50 0x7f298814d3f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.679 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f298de6f640 1 -- 172.21.15.183:0/2092898849 >> 172.21.15.183:0/2092898849 conn(0x7f2988065ea0 msgr2=0x7f29880662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:10.679 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f298de6f640 1 -- 172.21.15.183:0/2092898849 shutdown_connections 2023-10-30T20:23:10.679 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.662+0000 7f298de6f640 1 -- 172.21.15.183:0/2092898849 wait complete. 2023-10-30T20:23:10.679 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.663+0000 7f298de6f640 10 monclient: build_initial_monmap 2023-10-30T20:23:10.680 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.663+0000 7f298de6f640 10 monclient: monmap: 2023-10-30T20:23:10.680 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:10.680 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:10.680 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2023-10-30T20:23:10.680 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2023-10-30T20:23:10.680 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:10.681 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2023-10-30T20:23:10.681 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:10.681 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:10.681 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:10.681 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:10.682 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.663+0000 7f298de6f640 1 librados: starting msgr at 2023-10-30T20:23:10.682 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.663+0000 7f298de6f640 1 librados: starting objecter 2023-10-30T20:23:10.682 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.663+0000 7f298de6f640 1 Processor -- start 2023-10-30T20:23:10.682 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 1 -- start start 2023-10-30T20:23:10.682 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 1 librados: setting wanted keys 2023-10-30T20:23:10.683 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 1 librados: calling monclient init 2023-10-30T20:23:10.683 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 10 monclient: init 2023-10-30T20:23:10.683 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:10.683 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T20:23:10.683 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29880b4440 0x7f298813bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:10.684 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 10 monclient(hunting): picked mon.noname-a con 0x7f29880b4440 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:10.684 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f298813ed50 0x7f29880ac7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:10.684 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 10 monclient(hunting): picked mon.noname-c con 0x7f298813ed50 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:10.684 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29881455b0 0x7f29880b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:10.684 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 10 monclient(hunting): picked mon.noname-b con 0x7f29881455b0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:10.685 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:10.685 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29880b2c70 con 0x7f29880b4440 2023-10-30T20:23:10.685 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:10.685 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2988144250 con 0x7f29881455b0 2023-10-30T20:23:10.685 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:10.686 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29880afe30 con 0x7f298813ed50 2023-10-30T20:23:10.686 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:10.686 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.664+0000 7f298de6f640 10 monclient(hunting): authenticate will time out at 1509.866596s 2023-10-30T20:23:10.686 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f2987fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29881455b0 0x7f29880b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:10.686 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f29877fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29880b4440 0x7f298813bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:10.687 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f2986ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f298813ed50 0x7f29880ac7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:10.687 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f29877fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29880b4440 0x7f298813bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.183:55970/0 (socket says 172.21.15.183:55970) 2023-10-30T20:23:10.687 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f29877fe640 1 -- 172.21.15.183:0/3407256101 learned_addr learned my addr 172.21.15.183:0/3407256101 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-10-30T20:23:10.687 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f29877fe640 10 monclient(hunting): get_auth_request con 0x7f29880b4440 auth_method 0 2023-10-30T20:23:10.687 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f29877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:10.687 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f29877fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:10.688 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f29877fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:10.688 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f2987fff640 10 monclient(hunting): get_auth_request con 0x7f29881455b0 auth_method 0 2023-10-30T20:23:10.688 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f2987fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:10.688 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f2987fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:10.688 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f2987fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:10.689 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f2986ffd640 10 monclient(hunting): get_auth_request con 0x7f298813ed50 auth_method 0 2023-10-30T20:23:10.689 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f2986ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:10.689 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f2986ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:10.689 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f2986ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:10.689 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f29877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:10.689 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f29877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:10.690 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f29877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:10.690 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f2987fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:10.690 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.665+0000 7f2987fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:10.690 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.666+0000 7f2987fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:10.690 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.666+0000 7f2986ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:10.691 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.666+0000 7f2986ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:10.691 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.666+0000 7f2986ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:10.691 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.668+0000 7f29877fe640 10 monclient(hunting): handle_auth_done global_id 4134 payload 824 2023-10-30T20:23:10.691 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.668+0000 7f29877fe640 1 -- 172.21.15.183:0/3407256101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f298813ed50 msgr2=0x7f29880ac7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:10.691 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.668+0000 7f29877fe640 1 --2- 172.21.15.183:0/3407256101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f298813ed50 0x7f29880ac7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.691 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.668+0000 7f29877fe640 1 -- 172.21.15.183:0/3407256101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29881455b0 msgr2=0x7f29880b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:10.692 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.668+0000 7f29877fe640 1 --2- 172.21.15.183:0/3407256101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29881455b0 0x7f29880b0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.692 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.668+0000 7f29877fe640 10 monclient: _finish_hunting 0 2023-10-30T20:23:10.692 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.668+0000 7f29877fe640 1 monclient: found mon.noname-a 2023-10-30T20:23:10.692 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.668+0000 7f29877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:10.692 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.668+0000 7f29877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:23:10.693 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.668+0000 7f29877fe640 1 -- 172.21.15.183:0/3407256101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29880ad3d0 con 0x7f29880b4440 2023-10-30T20:23:10.693 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.668+0000 7f2987fff640 1 --2- 172.21.15.183:0/3407256101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29881455b0 0x7f29880b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T20:23:10.695 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.670+0000 7f29877fe640 1 --2- 172.21.15.183:0/3407256101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29880b4440 0x7f298813bad0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f297800c730 tx=0x7f297800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:10.695 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.670+0000 7f296ffff640 10 client.?.objecter ms_handle_connect 0x7f29880b4440 2023-10-30T20:23:10.695 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.670+0000 7f296ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:10.695 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f296ffff640 1 -- 172.21.15.183:0/3407256101 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978005ab0 con 0x7f29880b4440 2023-10-30T20:23:10.695 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f296ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:10.696 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f296ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:23:10.696 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f296ffff640 10 monclient: dump: 2023-10-30T20:23:10.696 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:10.696 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:10.697 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.697 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.697 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:10.697 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:10.697 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:10.698 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:10.698 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:10.698 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:10.698 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f296ffff640 10 monclient: _finish_auth 0 2023-10-30T20:23:10.698 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f296ffff640 10 monclient: _check_auth_tickets 2023-10-30T20:23:10.699 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f296ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:10.699 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f296ffff640 1 -- 172.21.15.183:0/3407256101 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297800e040 con 0x7f29880b4440 2023-10-30T20:23:10.699 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f298de6f640 5 monclient: authenticate success, global_id 4134 2023-10-30T20:23:10.699 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f298de6f640 10 monclient: _renew_subs 2023-10-30T20:23:10.699 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f298de6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:10.700 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f298de6f640 1 -- 172.21.15.183:0/3407256101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298806dca0 con 0x7f29880b4440 2023-10-30T20:23:10.700 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f296ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:10.700 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f296ffff640 1 -- 172.21.15.183:0/3407256101 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978005e60 con 0x7f29880b4440 2023-10-30T20:23:10.700 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f296ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:10.700 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f296ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:23:10.700 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f296ffff640 10 monclient: dump: 2023-10-30T20:23:10.701 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:10.701 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:10.701 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.701 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:10.701 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:10.702 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:10.702 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:10.702 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:10.702 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:10.702 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:10.703 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f2985ffb640 4 set_mon_vals no callback set 2023-10-30T20:23:10.703 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f298de6f640 10 client.4134.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:10.703 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f298de6f640 10 monclient: _renew_subs 2023-10-30T20:23:10.703 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f298de6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:10.703 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f298de6f640 1 -- 172.21.15.183:0/3407256101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29880b0b50 con 0x7f29880b4440 2023-10-30T20:23:10.704 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.674+0000 7f298de6f640 1 librados: init done 2023-10-30T20:23:10.704 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.675+0000 7f298de6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:10.704 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.675+0000 7f298de6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:10.704 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.675+0000 7f298de6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:10.704 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.675+0000 7f298de6f640 1 -- 172.21.15.183:0/3407256101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f298804e9b0 con 0x7f29880b4440 2023-10-30T20:23:10.770 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.769+0000 7f296ffff640 1 -- 172.21.15.183:0/3407256101 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2978007370 con 0x7f29880b4440 2023-10-30T20:23:10.774 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.773+0000 7f296ffff640 1 -- 172.21.15.183:0/3407256101 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f297800ac00 con 0x7f29880b4440 2023-10-30T20:23:10.774 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.773+0000 7f296ffff640 10 client.4134.objecter ms_dispatch 0x7f2988000b90 osd_map(6..6 src has 1..6) v4 2023-10-30T20:23:10.774 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.773+0000 7f296ffff640 3 client.4134.objecter handle_osd_map got epochs [6,6] > 0 2023-10-30T20:23:10.775 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.773+0000 7f296ffff640 3 client.4134.objecter handle_osd_map decoding full epoch 6 2023-10-30T20:23:10.775 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.773+0000 7f296ffff640 20 client.4134.objecter dump_active .. 0 homeless 2023-10-30T20:23:10.775 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.773+0000 7f296ffff640 1 -- 172.21.15.183:0/3407256101 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f297800add0 con 0x7f29880b4440 2023-10-30T20:23:10.775 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.773+0000 7f296ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:10.775 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.773+0000 7f296ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:10.846 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.844+0000 7f298de6f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "678fc58e-ed6a-4fd0-9825-93297e3af6f6", "id": 4}] 2023-10-30T20:23:10.846 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.845+0000 7f298de6f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "678fc58e-ed6a-4fd0-9825-93297e3af6f6", "id": 4}] 2023-10-30T20:23:10.846 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.845+0000 7f298de6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:10.846 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.845+0000 7f298de6f640 1 -- 172.21.15.183:0/3407256101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "678fc58e-ed6a-4fd0-9825-93297e3af6f6", "id": 4} v 0) v1 -- 0x7f298813f4a0 con 0x7f29880b4440 2023-10-30T20:23:10.898 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.897+0000 7f296ffff640 1 -- 172.21.15.183:0/3407256101 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "678fc58e-ed6a-4fd0-9825-93297e3af6f6", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f29780077b0 con 0x7f29880b4440 2023-10-30T20:23:10.898 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.897+0000 7f296ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "678fc58e-ed6a-4fd0-9825-93297e3af6f6", "id": 4}] 2023-10-30T20:23:10.899 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.897+0000 7f296ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:10.899 INFO:teuthology.orchestra.run.smithi183.stdout:4 2023-10-30T20:23:10.900 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 10 librados: watch_flush enter 2023-10-30T20:23:10.900 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 10 librados: watch_flush exit 2023-10-30T20:23:10.901 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 20 client.4134.objecter shutdown clearing up homeless session... 2023-10-30T20:23:10.901 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 10 client.4134.objecter successfully canceled tick 2023-10-30T20:23:10.901 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 10 monclient: shutdown 2023-10-30T20:23:10.901 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:10.901 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 1 -- 172.21.15.183:0/3407256101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29880b4440 msgr2=0x7f298813bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:10.902 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 1 --2- 172.21.15.183:0/3407256101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29880b4440 0x7f298813bad0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f297800c730 tx=0x7f297800cbf0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.902 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 1 -- 172.21.15.183:0/3407256101 shutdown_connections 2023-10-30T20:23:10.902 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 1 --2- 172.21.15.183:0/3407256101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29881455b0 0x7f29880b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.902 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 1 --2- 172.21.15.183:0/3407256101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f298813ed50 0x7f29880ac7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.902 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 1 --2- 172.21.15.183:0/3407256101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29880b4440 0x7f298813bad0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:10.903 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 1 -- 172.21.15.183:0/3407256101 >> 172.21.15.183:0/3407256101 conn(0x7f2988065ea0 msgr2=0x7f29880b1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:10.903 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 1 -- 172.21.15.183:0/3407256101 shutdown_connections 2023-10-30T20:23:10.903 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 1 -- 172.21.15.183:0/3407256101 wait complete. 2023-10-30T20:23:10.903 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:10.898+0000 7f298de6f640 1 librados: shutdown 2023-10-30T20:23:10.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new f945159a-a53c-48ba-97df-9374f9e56ffd 5 2023-10-30T20:23:11.007 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.006+0000 7f822ecc9640 10 monclient: get_monmap_and_config 2023-10-30T20:23:11.007 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.006+0000 7f822ecc9640 10 monclient: build_initial_monmap 2023-10-30T20:23:11.007 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.006+0000 7f822ecc9640 10 monclient: monmap: 2023-10-30T20:23:11.008 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:11.008 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.008 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:23:11.007056+0000 2023-10-30T20:23:11.008 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:23:11.007056+0000 2023-10-30T20:23:11.008 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:11.009 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.009 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:11.009 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:11.009 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:11.009 INFO:teuthology.orchestra.run.smithi183.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:11.010 INFO:teuthology.orchestra.run.smithi183.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:11.010 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.012 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.006+0000 7f822ecc9640 1 Processor -- start 2023-10-30T20:23:11.012 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.006+0000 7f822ecc9640 1 -- start start 2023-10-30T20:23:11.013 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.006+0000 7f822ecc9640 10 monclient: init 2023-10-30T20:23:11.013 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.007+0000 7f822ecc9640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:11.013 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.007+0000 7f822ecc9640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2023-10-30T20:23:11.013 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.007+0000 7f822ecc9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f82280b4440 0x7f8228144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:11.013 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.007+0000 7f822ecc9640 10 monclient(hunting): picked mon.noname-c con 0x7f82280b4440 addr v2:172.21.15.183:3300/0 2023-10-30T20:23:11.014 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.007+0000 7f822ecc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82281453e0 addr v1:172.21.15.183:6789/0 2023-10-30T20:23:11.014 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.007+0000 7f822ecc9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82280b2230 addr v1:172.21.15.69:6789/0 2023-10-30T20:23:11.014 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.007+0000 7f822ecc9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82280a9da0 con 0x7f82280b2230 2023-10-30T20:23:11.014 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.007+0000 7f822ecc9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82280aa080 con 0x7f82281453e0 2023-10-30T20:23:11.014 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.007+0000 7f822ecc9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:11.015 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.007+0000 7f822ecc9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8228067a10 con 0x7f82280b4440 2023-10-30T20:23:11.015 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.007+0000 7f822ecc9640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:11.015 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.007+0000 7f822ecc9640 10 monclient(hunting): authenticate will time out at 1510.209531s 2023-10-30T20:23:11.015 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.008+0000 7f822d266640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f82280b2230 0x7f822814c3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.183:36820/0 (socket says 172.21.15.183:36820) 2023-10-30T20:23:11.015 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.008+0000 7f822d266640 1 -- 172.21.15.183:0/3864942995 learned_addr learned my addr 172.21.15.183:0/3864942995 (peer_addr_for_me v1:172.21.15.183:0/0) 2023-10-30T20:23:11.015 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.008+0000 7f822ca65640 1 --2- 172.21.15.183:0/3864942995 >> v2:172.21.15.183:3300/0 conn(0x7f82280b4440 0x7f8228144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:11.016 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.009+0000 7f822ca65640 10 monclient(hunting): get_auth_request con 0x7f82280b4440 auth_method 0 2023-10-30T20:23:11.016 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.009+0000 7f822ca65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:11.016 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.009+0000 7f822ca65640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:11.016 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.009+0000 7f822ca65640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:11.016 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.010+0000 7f822ca65640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:11.017 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.010+0000 7f822ca65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:11.017 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.010+0000 7f822ca65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:11.017 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.010+0000 7f82277fe640 1 -- 172.21.15.183:0/3864942995 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061689146 0 0) 0x7f8220001490 con 0x7f82280b2230 2023-10-30T20:23:11.017 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.010+0000 7f82277fe640 10 monclient(hunting): _negotiate 2023-10-30T20:23:11.017 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.010+0000 7f82277fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:11.017 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.010+0000 7f82277fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:11.018 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.010+0000 7f82277fe640 10 monclient(hunting): my global_id is 4137 2023-10-30T20:23:11.018 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.010+0000 7f82277fe640 1 -- 172.21.15.183:0/3864942995 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8218003520 con 0x7f82280b2230 2023-10-30T20:23:11.019 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.012+0000 7f822ca65640 10 monclient(hunting): handle_auth_done global_id 4141 payload 274 2023-10-30T20:23:11.019 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.012+0000 7f822ca65640 1 -- 172.21.15.183:0/3864942995 >> v1:172.21.15.183:6789/0 conn(0x7f82281453e0 legacy=0x7f82280b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:11.019 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.012+0000 7f822ca65640 1 -- 172.21.15.183:0/3864942995 >> v1:172.21.15.69:6789/0 conn(0x7f82280b2230 legacy=0x7f822814c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:11.019 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.012+0000 7f822ca65640 10 monclient: _finish_hunting 0 2023-10-30T20:23:11.020 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.012+0000 7f822ca65640 1 monclient: found mon.noname-c 2023-10-30T20:23:11.020 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.012+0000 7f822ca65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:11.020 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.012+0000 7f822ca65640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:23:11.020 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.012+0000 7f822ca65640 1 -- 172.21.15.183:0/3864942995 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82280aa360 con 0x7f82280b4440 2023-10-30T20:23:11.020 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.015+0000 7f822ca65640 1 --2- 172.21.15.183:0/3864942995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82280b4440 0x7f8228144ea0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f821c0097a0 tx=0x7f821c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21c547393679676a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:11.022 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f82277fe640 1 -- 172.21.15.183:0/3864942995 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c0315a0 con 0x7f82280b4440 2023-10-30T20:23:11.023 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f82277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:11.023 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f82277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:23:11.023 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f82277fe640 10 monclient: dump: 2023-10-30T20:23:11.023 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:11.024 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.024 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.024 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.024 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:11.024 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.025 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:11.025 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:11.025 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:11.025 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.025 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f82277fe640 10 monclient: _finish_auth 0 2023-10-30T20:23:11.026 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f82277fe640 10 monclient: _check_auth_tickets 2023-10-30T20:23:11.026 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f82277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:11.026 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f82277fe640 1 -- 172.21.15.183:0/3864942995 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f821c031710 con 0x7f82280b4440 2023-10-30T20:23:11.026 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f82277fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:11.026 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f82277fe640 1 -- 172.21.15.183:0/3864942995 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c031ab0 con 0x7f82280b4440 2023-10-30T20:23:11.027 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f82277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:11.027 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f82277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:23:11.027 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f82277fe640 10 monclient: dump: 2023-10-30T20:23:11.027 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:11.027 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.028 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.028 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.028 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:11.028 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.028 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:11.029 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:11.029 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:11.029 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.029 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f822ecc9640 5 monclient: authenticate success, global_id 4141 2023-10-30T20:23:11.029 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f822ecc9640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:11.029 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.021+0000 7f822ecc9640 4 set_mon_vals no callback set 2023-10-30T20:23:11.033 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.022+0000 7f822ecc9640 10 monclient: shutdown 2023-10-30T20:23:11.034 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.022+0000 7f822ecc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:11.034 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.022+0000 7f822ecc9640 1 -- 172.21.15.183:0/3864942995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82280b4440 msgr2=0x7f8228144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:11.034 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.022+0000 7f822ecc9640 1 --2- 172.21.15.183:0/3864942995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82280b4440 0x7f8228144ea0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f821c0097a0 tx=0x7f821c02b840 comp rx=0 tx=0).stop 2023-10-30T20:23:11.034 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.022+0000 7f822ecc9640 1 -- 172.21.15.183:0/3864942995 shutdown_connections 2023-10-30T20:23:11.034 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.022+0000 7f822ecc9640 1 --2- 172.21.15.183:0/3864942995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82280b4440 0x7f8228144ea0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.035 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.022+0000 7f822ecc9640 1 -- 172.21.15.183:0/3864942995 >> 172.21.15.183:0/3864942995 conn(0x7f8228065ea0 msgr2=0x7f82280662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:11.035 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.022+0000 7f822ecc9640 1 -- 172.21.15.183:0/3864942995 shutdown_connections 2023-10-30T20:23:11.035 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.022+0000 7f822ecc9640 1 -- 172.21.15.183:0/3864942995 wait complete. 2023-10-30T20:23:11.035 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.023+0000 7f822ecc9640 10 monclient: build_initial_monmap 2023-10-30T20:23:11.035 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.023+0000 7f822ecc9640 10 monclient: monmap: 2023-10-30T20:23:11.036 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:11.036 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:11.036 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2023-10-30T20:23:11.036 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2023-10-30T20:23:11.036 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:11.037 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2023-10-30T20:23:11.037 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:11.037 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:11.037 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:11.037 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.038 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.024+0000 7f822ecc9640 1 librados: starting msgr at 2023-10-30T20:23:11.038 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.024+0000 7f822ecc9640 1 librados: starting objecter 2023-10-30T20:23:11.038 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.024+0000 7f822ecc9640 1 Processor -- start 2023-10-30T20:23:11.038 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.024+0000 7f822ecc9640 1 -- start start 2023-10-30T20:23:11.039 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.024+0000 7f822ecc9640 1 librados: setting wanted keys 2023-10-30T20:23:11.039 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.024+0000 7f822ecc9640 1 librados: calling monclient init 2023-10-30T20:23:11.039 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.024+0000 7f822ecc9640 10 monclient: init 2023-10-30T20:23:11.039 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:11.039 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T20:23:11.040 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82280b2230 0x7f8228140150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:11.040 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 10 monclient(hunting): picked mon.noname-a con 0x7f82280b2230 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:11.040 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82280b4440 0x7f82280af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:11.040 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 10 monclient(hunting): picked mon.noname-c con 0x7f82280b4440 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:11.040 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82281453e0 0x7f822813a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:11.041 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 10 monclient(hunting): picked mon.noname-b con 0x7f82281453e0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:11.041 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:11.041 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82280a8b50 con 0x7f82280b2230 2023-10-30T20:23:11.041 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:11.041 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8228140910 con 0x7f82281453e0 2023-10-30T20:23:11.042 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:11.042 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8228140bf0 con 0x7f82280b4440 2023-10-30T20:23:11.042 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:11.042 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822ecc9640 10 monclient(hunting): authenticate will time out at 1510.227365s 2023-10-30T20:23:11.042 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.025+0000 7f822d266640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82281453e0 0x7f822813a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:11.043 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f822ca65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82280b2230 0x7f8228140150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:11.043 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f8227fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82280b4440 0x7f82280af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:11.043 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f822d266640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82281453e0 0x7f822813a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:58438/0 (socket says 172.21.15.183:58438) 2023-10-30T20:23:11.043 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f822d266640 1 -- 172.21.15.183:0/4107548981 learned_addr learned my addr 172.21.15.183:0/4107548981 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-10-30T20:23:11.043 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f822d266640 10 monclient(hunting): get_auth_request con 0x7f82281453e0 auth_method 0 2023-10-30T20:23:11.044 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f822d266640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:11.044 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f822d266640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:11.044 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f822d266640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:11.044 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f822ca65640 10 monclient(hunting): get_auth_request con 0x7f82280b2230 auth_method 0 2023-10-30T20:23:11.044 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f822ca65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:11.044 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f822ca65640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:11.045 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f822ca65640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:11.045 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f8227fff640 10 monclient(hunting): get_auth_request con 0x7f82280b4440 auth_method 0 2023-10-30T20:23:11.045 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f8227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:11.045 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f8227fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:11.045 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f8227fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:11.045 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f822d266640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:11.046 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f822d266640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:11.046 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.026+0000 7f822d266640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:11.046 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.027+0000 7f822ca65640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:11.046 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.027+0000 7f822ca65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:11.046 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.027+0000 7f822ca65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:11.047 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.027+0000 7f8227fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:11.047 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.027+0000 7f8227fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:11.047 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.027+0000 7f8227fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:11.047 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.029+0000 7f822d266640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2023-10-30T20:23:11.047 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.029+0000 7f822d266640 1 -- 172.21.15.183:0/4107548981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82280b4440 msgr2=0x7f82280af2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:11.047 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.029+0000 7f822d266640 1 --2- 172.21.15.183:0/4107548981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82280b4440 0x7f82280af2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.048 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.029+0000 7f822d266640 1 -- 172.21.15.183:0/4107548981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82280b2230 msgr2=0x7f8228140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:11.048 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.029+0000 7f822d266640 1 --2- 172.21.15.183:0/4107548981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82280b2230 0x7f8228140150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.048 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.029+0000 7f822d266640 10 monclient: _finish_hunting 0 2023-10-30T20:23:11.048 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.029+0000 7f822d266640 1 monclient: found mon.noname-b 2023-10-30T20:23:11.049 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.029+0000 7f822d266640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:11.049 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.029+0000 7f822d266640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:23:11.049 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.029+0000 7f822d266640 1 -- 172.21.15.183:0/4107548981 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8228142540 con 0x7f82281453e0 2023-10-30T20:23:11.049 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.031+0000 7f822d266640 1 --2- 172.21.15.183:0/4107548981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82281453e0 0x7f822813a730 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f822000cb70 tx=0x7f822000a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:11.049 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.031+0000 7f82257fa640 10 client.?.objecter ms_handle_connect 0x7f82281453e0 2023-10-30T20:23:11.049 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.031+0000 7f82257fa640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:11.051 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f82257fa640 1 -- 172.21.15.183:0/4107548981 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822000ace0 con 0x7f82281453e0 2023-10-30T20:23:11.051 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f82257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:11.051 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f82257fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:23:11.051 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f82257fa640 10 monclient: dump: 2023-10-30T20:23:11.052 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:11.052 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.052 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.052 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.052 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:11.053 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.053 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:11.053 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:11.053 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:11.053 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.054 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f82257fa640 10 monclient: _finish_auth 0 2023-10-30T20:23:11.054 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f82257fa640 10 monclient: _check_auth_tickets 2023-10-30T20:23:11.054 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f82257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:11.054 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f822ecc9640 5 monclient: authenticate success, global_id 4144 2023-10-30T20:23:11.054 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f822ecc9640 10 monclient: _renew_subs 2023-10-30T20:23:11.055 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f822ecc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:11.055 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f822ecc9640 1 -- 172.21.15.183:0/4107548981 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822813f7a0 con 0x7f82281453e0 2023-10-30T20:23:11.055 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f822ecc9640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:11.055 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f822ecc9640 10 monclient: _renew_subs 2023-10-30T20:23:11.055 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f822ecc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:11.056 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f822ecc9640 1 -- 172.21.15.183:0/4107548981 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82280b0fa0 con 0x7f82281453e0 2023-10-30T20:23:11.056 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.033+0000 7f822ecc9640 1 librados: init done 2023-10-30T20:23:11.056 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.034+0000 7f822ecc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:11.056 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.034+0000 7f822ecc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:11.056 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.034+0000 7f822ecc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:11.057 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.034+0000 7f822ecc9640 1 -- 172.21.15.183:0/4107548981 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8228140da0 con 0x7f82281453e0 2023-10-30T20:23:11.057 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.035+0000 7f82257fa640 1 -- 172.21.15.183:0/4107548981 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822000e040 con 0x7f82281453e0 2023-10-30T20:23:11.057 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.035+0000 7f82257fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:11.057 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.035+0000 7f82257fa640 1 -- 172.21.15.183:0/4107548981 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822001c640 con 0x7f82281453e0 2023-10-30T20:23:11.057 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.035+0000 7f82257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:11.058 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.035+0000 7f82257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:23:11.058 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.035+0000 7f8226ffd640 4 set_mon_vals no callback set 2023-10-30T20:23:11.058 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.035+0000 7f82257fa640 10 monclient: dump: 2023-10-30T20:23:11.058 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:11.058 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.059 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.059 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.059 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:11.059 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.061 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:11.061 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:11.061 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:11.061 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.137 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.135+0000 7f82257fa640 1 -- 172.21.15.183:0/4107548981 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f822001cb50 con 0x7f82281453e0 2023-10-30T20:23:11.137 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.135+0000 7f82257fa640 1 -- 172.21.15.183:0/4107548981 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f8220012a90 con 0x7f82281453e0 2023-10-30T20:23:11.137 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.135+0000 7f82257fa640 10 client.4144.objecter ms_dispatch 0x7f8228000b90 osd_map(7..7 src has 1..7) v4 2023-10-30T20:23:11.138 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.135+0000 7f82257fa640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2023-10-30T20:23:11.138 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.135+0000 7f82257fa640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2023-10-30T20:23:11.138 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.135+0000 7f82257fa640 20 client.4144.objecter dump_active .. 0 homeless 2023-10-30T20:23:11.138 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.135+0000 7f82257fa640 1 -- 172.21.15.183:0/4107548981 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f8220012dd0 con 0x7f82281453e0 2023-10-30T20:23:11.138 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.136+0000 7f82257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:11.139 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.136+0000 7f82257fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:11.209 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.208+0000 7f822ecc9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f945159a-a53c-48ba-97df-9374f9e56ffd", "id": 5}] 2023-10-30T20:23:11.209 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.208+0000 7f822ecc9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f945159a-a53c-48ba-97df-9374f9e56ffd", "id": 5}] 2023-10-30T20:23:11.210 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.208+0000 7f822ecc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:11.210 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.208+0000 7f822ecc9640 1 -- 172.21.15.183:0/4107548981 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f945159a-a53c-48ba-97df-9374f9e56ffd", "id": 5} v 0) v1 -- 0x7f822804e9b0 con 0x7f82281453e0 2023-10-30T20:23:11.280 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.279+0000 7f82257fa640 1 -- 172.21.15.183:0/4107548981 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f945159a-a53c-48ba-97df-9374f9e56ffd", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8220012400 con 0x7f82281453e0 2023-10-30T20:23:11.281 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.279+0000 7f82257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f945159a-a53c-48ba-97df-9374f9e56ffd", "id": 5}] 2023-10-30T20:23:11.281 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.279+0000 7f82257fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:11.281 INFO:teuthology.orchestra.run.smithi183.stdout:5 2023-10-30T20:23:11.283 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.280+0000 7f822ecc9640 10 librados: watch_flush enter 2023-10-30T20:23:11.283 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.280+0000 7f822ecc9640 10 librados: watch_flush exit 2023-10-30T20:23:11.283 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.280+0000 7f822ecc9640 20 client.4144.objecter shutdown clearing up homeless session... 2023-10-30T20:23:11.283 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.280+0000 7f822ecc9640 10 client.4144.objecter successfully canceled tick 2023-10-30T20:23:11.284 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.280+0000 7f822ecc9640 10 monclient: shutdown 2023-10-30T20:23:11.284 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.280+0000 7f822ecc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:11.284 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.280+0000 7f822ecc9640 1 -- 172.21.15.183:0/4107548981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82281453e0 msgr2=0x7f822813a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:11.284 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.280+0000 7f822ecc9640 1 --2- 172.21.15.183:0/4107548981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82281453e0 0x7f822813a730 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f822000cb70 tx=0x7f822000a430 comp rx=0 tx=0).stop 2023-10-30T20:23:11.284 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.280+0000 7f822ecc9640 1 -- 172.21.15.183:0/4107548981 shutdown_connections 2023-10-30T20:23:11.285 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.280+0000 7f822ecc9640 1 --2- 172.21.15.183:0/4107548981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82281453e0 0x7f822813a730 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.285 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.280+0000 7f822ecc9640 1 --2- 172.21.15.183:0/4107548981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82280b4440 0x7f82280af2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.285 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.280+0000 7f822ecc9640 1 --2- 172.21.15.183:0/4107548981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82280b2230 0x7f8228140150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.285 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.280+0000 7f822ecc9640 1 -- 172.21.15.183:0/4107548981 >> 172.21.15.183:0/4107548981 conn(0x7f8228065ea0 msgr2=0x7f82280b78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:11.285 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.280+0000 7f822ecc9640 1 -- 172.21.15.183:0/4107548981 shutdown_connections 2023-10-30T20:23:11.286 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.281+0000 7f822ecc9640 1 -- 172.21.15.183:0/4107548981 wait complete. 2023-10-30T20:23:11.286 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.281+0000 7f822ecc9640 1 librados: shutdown 2023-10-30T20:23:11.292 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 8184f59d-15e2-4bdc-90e3-fe1e231ca125 6 2023-10-30T20:23:11.327 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:11.326+0000 7fea71613040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T20:23:11.392 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.391+0000 7fafbf59e640 10 monclient: get_monmap_and_config 2023-10-30T20:23:11.393 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.391+0000 7fafbf59e640 10 monclient: build_initial_monmap 2023-10-30T20:23:11.393 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.391+0000 7fafbf59e640 10 monclient: monmap: 2023-10-30T20:23:11.393 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:11.393 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.393 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:23:11.392516+0000 2023-10-30T20:23:11.394 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:23:11.392516+0000 2023-10-30T20:23:11.394 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:11.394 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.394 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:11.394 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:11.395 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:11.395 INFO:teuthology.orchestra.run.smithi183.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:11.395 INFO:teuthology.orchestra.run.smithi183.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:11.395 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.397 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.392+0000 7fafbf59e640 1 Processor -- start 2023-10-30T20:23:11.397 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.392+0000 7fafbf59e640 1 -- start start 2023-10-30T20:23:11.397 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.392+0000 7fafbf59e640 10 monclient: init 2023-10-30T20:23:11.398 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.392+0000 7fafbf59e640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:11.398 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.392+0000 7fafbf59e640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-10-30T20:23:11.398 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.392+0000 7fafbf59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafc00b4250 0x7fafc00a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:11.398 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.393+0000 7fafbf59e640 10 monclient(hunting): picked mon.noname-b con 0x7fafc00b4250 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:11.398 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.393+0000 7fafbf59e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fafc013eb30 0x7fafc013ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:11.399 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.393+0000 7fafbf59e640 10 monclient(hunting): picked mon.noname-c con 0x7fafc013eb30 addr v2:172.21.15.183:3300/0 2023-10-30T20:23:11.399 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.393+0000 7fafbf59e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fafc0141910 addr v1:172.21.15.183:6789/0 2023-10-30T20:23:11.399 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.393+0000 7fafbf59e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc00a8b80 con 0x7fafc0141910 2023-10-30T20:23:11.399 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.393+0000 7fafbf59e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:11.399 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.393+0000 7fafbf59e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc0067880 con 0x7fafc013eb30 2023-10-30T20:23:11.400 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.393+0000 7fafbf59e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:11.400 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.393+0000 7fafbf59e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc00b4630 con 0x7fafc00b4250 2023-10-30T20:23:11.400 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.393+0000 7fafbf59e640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:11.400 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.393+0000 7fafbf59e640 10 monclient(hunting): authenticate will time out at 1510.595100s 2023-10-30T20:23:11.400 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.393+0000 7fafbdd9b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fafc013eb30 0x7fafc013ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:11.401 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.393+0000 7fafbe59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafc00b4250 0x7fafc00a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:11.401 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.394+0000 7fafbed9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fafc0141910 0x7fafc0141da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:53866/0 (socket says 172.21.15.183:53866) 2023-10-30T20:23:11.401 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.394+0000 7fafbed9d640 1 -- 172.21.15.183:0/4028059904 learned_addr learned my addr 172.21.15.183:0/4028059904 (peer_addr_for_me v1:172.21.15.183:0/0) 2023-10-30T20:23:11.401 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.394+0000 7fafbe59c640 10 monclient(hunting): get_auth_request con 0x7fafc00b4250 auth_method 0 2023-10-30T20:23:11.401 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.394+0000 7fafbe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:11.402 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.394+0000 7fafbe59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:11.402 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.394+0000 7fafbe59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:11.402 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.394+0000 7fafbdd9b640 10 monclient(hunting): get_auth_request con 0x7fafc013eb30 auth_method 0 2023-10-30T20:23:11.402 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.394+0000 7fafbdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:11.402 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.394+0000 7fafbdd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:11.403 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.394+0000 7fafbdd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:11.403 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.394+0000 7fafbe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:11.403 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.394+0000 7fafbe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:11.403 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.394+0000 7fafbe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:11.403 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.395+0000 7fafbdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:11.403 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.395+0000 7fafbdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:11.404 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.395+0000 7fafbdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:11.404 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.396+0000 7fafbdd9b640 10 monclient(hunting): handle_auth_done global_id 4147 payload 274 2023-10-30T20:23:11.405 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.396+0000 7fafbdd9b640 1 -- 172.21.15.183:0/4028059904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafc00b4250 msgr2=0x7fafc00a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:11.405 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.396+0000 7fafbdd9b640 1 --2- 172.21.15.183:0/4028059904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafc00b4250 0x7fafc00a83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.405 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.396+0000 7fafbdd9b640 1 -- 172.21.15.183:0/4028059904 >> v1:172.21.15.183:6789/0 conn(0x7fafc0141910 legacy=0x7fafc0141da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:11.405 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.396+0000 7fafbdd9b640 10 monclient: _finish_hunting 0 2023-10-30T20:23:11.405 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.396+0000 7fafbdd9b640 1 monclient: found mon.noname-c 2023-10-30T20:23:11.406 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.396+0000 7fafbdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:11.406 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.396+0000 7fafbdd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:23:11.406 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.396+0000 7fafbdd9b640 1 -- 172.21.15.183:0/4028059904 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc01424b0 con 0x7fafc013eb30 2023-10-30T20:23:11.406 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.401+0000 7fafbdd9b640 1 --2- 172.21.15.183:0/4028059904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafc013eb30 0x7fafc013ef50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fafb40097a0 tx=0x7fafb402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1500f9db1223a180 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:11.431 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbd59a640 1 -- 172.21.15.183:0/4028059904 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb40315a0 con 0x7fafc013eb30 2023-10-30T20:23:11.431 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:11.431 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:23:11.432 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbd59a640 10 monclient: dump: 2023-10-30T20:23:11.432 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:11.432 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.432 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.432 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.433 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:11.433 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.433 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:11.433 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:11.433 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:11.434 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.434 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbd59a640 10 monclient: _finish_auth 0 2023-10-30T20:23:11.434 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbd59a640 10 monclient: _check_auth_tickets 2023-10-30T20:23:11.434 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:11.434 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbf59e640 5 monclient: authenticate success, global_id 4147 2023-10-30T20:23:11.435 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbd59a640 1 -- 172.21.15.183:0/4028059904 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb4031710 con 0x7fafc013eb30 2023-10-30T20:23:11.435 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbd59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:11.435 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbf59e640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:11.435 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbf59e640 4 set_mon_vals no callback set 2023-10-30T20:23:11.435 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbd59a640 1 -- 172.21.15.183:0/4028059904 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb4031ab0 con 0x7fafc013eb30 2023-10-30T20:23:11.436 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:11.436 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:23:11.436 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbd59a640 10 monclient: dump: 2023-10-30T20:23:11.436 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:11.436 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.436 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.437 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.437 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:11.437 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.437 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:11.437 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:11.438 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:11.438 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.441 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbf59e640 10 monclient: shutdown 2023-10-30T20:23:11.441 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbf59e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:11.441 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbf59e640 1 -- 172.21.15.183:0/4028059904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafc013eb30 msgr2=0x7fafc013ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:11.441 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.430+0000 7fafbf59e640 1 --2- 172.21.15.183:0/4028059904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafc013eb30 0x7fafc013ef50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fafb40097a0 tx=0x7fafb402b840 comp rx=0 tx=0).stop 2023-10-30T20:23:11.441 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.431+0000 7fafbf59e640 1 -- 172.21.15.183:0/4028059904 shutdown_connections 2023-10-30T20:23:11.442 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.431+0000 7fafbf59e640 1 --2- 172.21.15.183:0/4028059904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafc013eb30 0x7fafc013ef50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.442 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.431+0000 7fafbf59e640 1 --2- 172.21.15.183:0/4028059904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafc00b4250 0x7fafc00a83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.442 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.431+0000 7fafbf59e640 1 -- 172.21.15.183:0/4028059904 >> 172.21.15.183:0/4028059904 conn(0x7fafc0065ea0 msgr2=0x7fafc00662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:11.442 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.431+0000 7fafbf59e640 1 -- 172.21.15.183:0/4028059904 shutdown_connections 2023-10-30T20:23:11.442 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.431+0000 7fafbf59e640 1 -- 172.21.15.183:0/4028059904 wait complete. 2023-10-30T20:23:11.443 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.431+0000 7fafbf59e640 10 monclient: build_initial_monmap 2023-10-30T20:23:11.443 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.431+0000 7fafbf59e640 10 monclient: monmap: 2023-10-30T20:23:11.443 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:11.443 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:11.443 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2023-10-30T20:23:11.444 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2023-10-30T20:23:11.444 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:11.444 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2023-10-30T20:23:11.444 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:11.444 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:11.445 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:11.445 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.445 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 1 librados: starting msgr at 2023-10-30T20:23:11.445 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 1 librados: starting objecter 2023-10-30T20:23:11.445 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 1 Processor -- start 2023-10-30T20:23:11.445 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 1 -- start start 2023-10-30T20:23:11.446 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 1 librados: setting wanted keys 2023-10-30T20:23:11.446 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 1 librados: calling monclient init 2023-10-30T20:23:11.446 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 10 monclient: init 2023-10-30T20:23:11.446 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:11.447 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T20:23:11.447 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafc00b4250 0x7fafc01cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:11.447 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 10 monclient(hunting): picked mon.noname-a con 0x7fafc00b4250 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:11.447 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafc013eb30 0x7fafc013cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:11.447 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 10 monclient(hunting): picked mon.noname-c con 0x7fafc013eb30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:11.447 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafc0141910 0x7fafc01d9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:11.448 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 10 monclient(hunting): picked mon.noname-b con 0x7fafc0141910 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:11.448 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:11.448 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc013c0d0 con 0x7fafc00b4250 2023-10-30T20:23:11.448 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:11.448 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc01382b0 con 0x7fafc0141910 2023-10-30T20:23:11.449 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:11.449 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc013c6f0 con 0x7fafc013eb30 2023-10-30T20:23:11.449 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:11.449 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.432+0000 7fafbf59e640 10 monclient(hunting): authenticate will time out at 1510.634813s 2023-10-30T20:23:11.449 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.434+0000 7fafbed9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafc0141910 0x7fafc01d9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:11.450 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.434+0000 7fafbed9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafc0141910 0x7fafc01d9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:58458/0 (socket says 172.21.15.183:58458) 2023-10-30T20:23:11.450 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.434+0000 7fafbed9d640 1 -- 172.21.15.183:0/1203435507 learned_addr learned my addr 172.21.15.183:0/1203435507 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-10-30T20:23:11.450 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.434+0000 7fafbed9d640 10 monclient(hunting): get_auth_request con 0x7fafc0141910 auth_method 0 2023-10-30T20:23:11.450 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.434+0000 7fafbed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:11.450 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.434+0000 7fafbed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:11.451 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.434+0000 7fafbed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:11.451 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.435+0000 7fafbed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:11.451 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.435+0000 7fafbed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:11.451 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.435+0000 7fafbed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:11.451 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.439+0000 7fafbe59c640 1 --2- 172.21.15.183:0/1203435507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafc00b4250 0x7fafc01cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:11.452 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.439+0000 7fafbdd9b640 1 --2- 172.21.15.183:0/1203435507 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafc013eb30 0x7fafc013cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:11.452 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.439+0000 7fafbed9d640 10 monclient(hunting): handle_auth_done global_id 4150 payload 824 2023-10-30T20:23:11.452 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.439+0000 7fafbed9d640 1 -- 172.21.15.183:0/1203435507 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafc013eb30 msgr2=0x7fafc013cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:11.452 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.439+0000 7fafbed9d640 1 --2- 172.21.15.183:0/1203435507 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafc013eb30 0x7fafc013cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.453 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.439+0000 7fafbed9d640 1 -- 172.21.15.183:0/1203435507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafc00b4250 msgr2=0x7fafc01cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:11.453 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.439+0000 7fafbed9d640 1 --2- 172.21.15.183:0/1203435507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafc00b4250 0x7fafc01cb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.453 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.439+0000 7fafbed9d640 10 monclient: _finish_hunting 0 2023-10-30T20:23:11.453 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.439+0000 7fafbed9d640 1 monclient: found mon.noname-b 2023-10-30T20:23:11.453 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.439+0000 7fafbed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:11.453 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.439+0000 7fafbed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:23:11.454 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.439+0000 7fafbed9d640 1 -- 172.21.15.183:0/1203435507 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc013d870 con 0x7fafc0141910 2023-10-30T20:23:11.455 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.445+0000 7fafbed9d640 1 --2- 172.21.15.183:0/1203435507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafc0141910 0x7fafc01d9d40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fafb800ca80 tx=0x7fafb800cf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:11.455 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.445+0000 7fafaeffd640 10 client.?.objecter ms_handle_connect 0x7fafc0141910 2023-10-30T20:23:11.455 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.445+0000 7fafaeffd640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:11.456 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafaeffd640 1 -- 172.21.15.183:0/1203435507 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb800ac30 con 0x7fafc0141910 2023-10-30T20:23:11.456 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:11.456 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafaeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:23:11.456 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafaeffd640 10 monclient: dump: 2023-10-30T20:23:11.456 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:11.457 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.457 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.457 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.457 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:11.457 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.458 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:11.458 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:11.458 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:11.458 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.458 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafaeffd640 10 monclient: _finish_auth 0 2023-10-30T20:23:11.458 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafaeffd640 10 monclient: _check_auth_tickets 2023-10-30T20:23:11.459 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:11.459 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafaeffd640 1 -- 172.21.15.183:0/1203435507 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb800e040 con 0x7fafc0141910 2023-10-30T20:23:11.459 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafaeffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:11.459 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafaeffd640 1 -- 172.21.15.183:0/1203435507 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8012400 con 0x7fafc0141910 2023-10-30T20:23:11.459 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafbf59e640 5 monclient: authenticate success, global_id 4150 2023-10-30T20:23:11.460 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafbf59e640 10 monclient: _renew_subs 2023-10-30T20:23:11.460 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafbf59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:11.460 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafbf59e640 1 -- 172.21.15.183:0/1203435507 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc013dad0 con 0x7fafc0141910 2023-10-30T20:23:11.460 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafbcd99640 4 set_mon_vals no callback set 2023-10-30T20:23:11.460 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:11.460 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafaeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:23:11.461 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafaeffd640 10 monclient: dump: 2023-10-30T20:23:11.461 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:11.461 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.461 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.461 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.462 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:11.462 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.462 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:11.462 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:11.462 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:11.463 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.463 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafbf59e640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:11.463 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafbf59e640 10 monclient: _renew_subs 2023-10-30T20:23:11.463 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafbf59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:11.463 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafbf59e640 1 -- 172.21.15.183:0/1203435507 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc01da460 con 0x7fafc0141910 2023-10-30T20:23:11.464 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.449+0000 7fafbf59e640 1 librados: init done 2023-10-30T20:23:11.464 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.450+0000 7fafbf59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:11.464 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.450+0000 7fafbf59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:11.464 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.450+0000 7fafbf59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:11.464 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.450+0000 7fafbf59e640 1 -- 172.21.15.183:0/1203435507 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc00619f0 con 0x7fafc0141910 2023-10-30T20:23:11.485 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:11.484+0000 7f594b411040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T20:23:11.551 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.550+0000 7fafaeffd640 1 -- 172.21.15.183:0/1203435507 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fafb8012870 con 0x7fafc0141910 2023-10-30T20:23:11.551 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.550+0000 7fafaeffd640 1 -- 172.21.15.183:0/1203435507 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fafb8007680 con 0x7fafc0141910 2023-10-30T20:23:11.552 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.550+0000 7fafaeffd640 10 client.4150.objecter ms_dispatch 0x7fafc0000b90 osd_map(8..8 src has 1..8) v4 2023-10-30T20:23:11.552 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.550+0000 7fafaeffd640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2023-10-30T20:23:11.552 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.550+0000 7fafaeffd640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2023-10-30T20:23:11.552 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.550+0000 7fafaeffd640 20 client.4150.objecter dump_active .. 0 homeless 2023-10-30T20:23:11.552 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.550+0000 7fafaeffd640 1 -- 172.21.15.183:0/1203435507 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fafb8007920 con 0x7fafc0141910 2023-10-30T20:23:11.553 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.550+0000 7fafaeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:11.553 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.550+0000 7fafaeffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:11.604 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:11.603+0000 7fea71613040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T20:23:11.621 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.619+0000 7fafbf59e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8184f59d-15e2-4bdc-90e3-fe1e231ca125", "id": 6}] 2023-10-30T20:23:11.621 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.619+0000 7fafbf59e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8184f59d-15e2-4bdc-90e3-fe1e231ca125", "id": 6}] 2023-10-30T20:23:11.621 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.619+0000 7fafbf59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:11.621 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.619+0000 7fafbf59e640 1 -- 172.21.15.183:0/1203435507 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8184f59d-15e2-4bdc-90e3-fe1e231ca125", "id": 6} v 0) v1 -- 0x7fafc013c8c0 con 0x7fafc0141910 2023-10-30T20:23:11.666 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:11.665+0000 7fea71613040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T20:23:11.689 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.688+0000 7fafaeffd640 1 -- 172.21.15.183:0/1203435507 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8184f59d-15e2-4bdc-90e3-fe1e231ca125", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fafb801c440 con 0x7fafc0141910 2023-10-30T20:23:11.689 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.688+0000 7fafaeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8184f59d-15e2-4bdc-90e3-fe1e231ca125", "id": 6}] 2023-10-30T20:23:11.689 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.688+0000 7fafaeffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:11.690 INFO:teuthology.orchestra.run.smithi183.stdout:6 2023-10-30T20:23:11.691 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 10 librados: watch_flush enter 2023-10-30T20:23:11.691 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 10 librados: watch_flush exit 2023-10-30T20:23:11.692 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 20 client.4150.objecter shutdown clearing up homeless session... 2023-10-30T20:23:11.692 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 10 client.4150.objecter successfully canceled tick 2023-10-30T20:23:11.692 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 10 monclient: shutdown 2023-10-30T20:23:11.692 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:11.693 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 1 -- 172.21.15.183:0/1203435507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafc0141910 msgr2=0x7fafc01d9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:11.693 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 1 --2- 172.21.15.183:0/1203435507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafc0141910 0x7fafc01d9d40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fafb800ca80 tx=0x7fafb800cf40 comp rx=0 tx=0).stop 2023-10-30T20:23:11.693 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 1 -- 172.21.15.183:0/1203435507 shutdown_connections 2023-10-30T20:23:11.693 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 1 --2- 172.21.15.183:0/1203435507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafc0141910 0x7fafc01d9d40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.693 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 1 --2- 172.21.15.183:0/1203435507 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafc013eb30 0x7fafc013cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.694 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 1 --2- 172.21.15.183:0/1203435507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafc00b4250 0x7fafc01cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.694 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 1 -- 172.21.15.183:0/1203435507 >> 172.21.15.183:0/1203435507 conn(0x7fafc0065ea0 msgr2=0x7fafc00b2170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:11.694 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 1 -- 172.21.15.183:0/1203435507 shutdown_connections 2023-10-30T20:23:11.694 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 1 -- 172.21.15.183:0/1203435507 wait complete. 2023-10-30T20:23:11.694 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.689+0000 7fafbf59e640 1 librados: shutdown 2023-10-30T20:23:11.700 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d 7 2023-10-30T20:23:11.766 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:11.764+0000 7f594b411040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T20:23:11.769 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:11.767+0000 7fea71613040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T20:23:11.804 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.803+0000 7f89b4ba9640 10 monclient: get_monmap_and_config 2023-10-30T20:23:11.805 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.803+0000 7f89b4ba9640 10 monclient: build_initial_monmap 2023-10-30T20:23:11.805 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.803+0000 7f89b4ba9640 10 monclient: monmap: 2023-10-30T20:23:11.805 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:11.805 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.806 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:23:11.804484+0000 2023-10-30T20:23:11.806 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:23:11.804484+0000 2023-10-30T20:23:11.806 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:11.806 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.806 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:11.807 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:11.807 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:11.807 INFO:teuthology.orchestra.run.smithi183.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:11.807 INFO:teuthology.orchestra.run.smithi183.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:11.807 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.809 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 1 Processor -- start 2023-10-30T20:23:11.810 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 1 -- start start 2023-10-30T20:23:11.810 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 10 monclient: init 2023-10-30T20:23:11.810 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:11.810 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2023-10-30T20:23:11.811 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f89b00b4440 0x7f89b0144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:11.811 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 10 monclient(hunting): picked mon.noname-a con 0x7f89b00b4440 addr v2:172.21.15.69:3300/0 2023-10-30T20:23:11.811 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f89b01453e0 addr v1:172.21.15.183:6789/0 2023-10-30T20:23:11.811 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f89b00b2230 0x7f89b014c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:11.811 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 10 monclient(hunting): picked mon.noname-c con 0x7f89b00b2230 addr v2:172.21.15.183:3300/0 2023-10-30T20:23:11.811 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b00a9da0 con 0x7f89b01453e0 2023-10-30T20:23:11.812 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:11.812 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b0067a10 con 0x7f89b00b4440 2023-10-30T20:23:11.812 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:11.812 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b00aa080 con 0x7f89b00b2230 2023-10-30T20:23:11.812 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:11.813 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.804+0000 7f89b4ba9640 10 monclient(hunting): authenticate will time out at 1511.006814s 2023-10-30T20:23:11.813 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.805+0000 7f89ae59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f89b00b4440 0x7f89b0144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:11.813 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.805+0000 7f89ae59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f89b00b4440 0x7f89b0144ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.183:56004/0 (socket says 172.21.15.183:56004) 2023-10-30T20:23:11.813 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.805+0000 7f89ae59c640 1 -- 172.21.15.183:0/290753097 learned_addr learned my addr 172.21.15.183:0/290753097 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-10-30T20:23:11.813 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.805+0000 7f89ae59c640 10 monclient(hunting): get_auth_request con 0x7f89b00b4440 auth_method 0 2023-10-30T20:23:11.814 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.805+0000 7f89ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:11.814 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.805+0000 7f89ae59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:11.814 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.805+0000 7f89ae59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:11.814 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.806+0000 7f89ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:11.814 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.806+0000 7f89ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:11.815 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.806+0000 7f89ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:11.815 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.807+0000 7f89ae59c640 10 monclient(hunting): handle_auth_done global_id 4143 payload 274 2023-10-30T20:23:11.815 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.807+0000 7f89ae59c640 1 -- 172.21.15.183:0/290753097 >> v2:172.21.15.183:3300/0 conn(0x7f89b00b2230 msgr2=0x7f89b014c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:11.815 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.807+0000 7f89ae59c640 1 --2- 172.21.15.183:0/290753097 >> v2:172.21.15.183:3300/0 conn(0x7f89b00b2230 0x7f89b014c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.815 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.807+0000 7f89ae59c640 1 -- 172.21.15.183:0/290753097 >> v1:172.21.15.183:6789/0 conn(0x7f89b01453e0 legacy=0x7f89b00b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:11.816 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.807+0000 7f89ae59c640 10 monclient: _finish_hunting 0 2023-10-30T20:23:11.816 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.807+0000 7f89ae59c640 1 monclient: found mon.noname-a 2023-10-30T20:23:11.816 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.807+0000 7f89ae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:11.816 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.807+0000 7f89ae59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:23:11.816 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.807+0000 7f89ae59c640 1 -- 172.21.15.183:0/290753097 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b00aa360 con 0x7f89b00b4440 2023-10-30T20:23:11.820 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.809+0000 7f89ae59c640 1 --2- 172.21.15.183:0/290753097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b00b4440 0x7f89b0144ea0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f89b00b1780 tx=0x7f89a002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccdd2c346591074e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:11.820 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89add9b640 1 -- 172.21.15.183:0/290753097 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a0031750 con 0x7f89b00b4440 2023-10-30T20:23:11.820 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:11.820 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89add9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:23:11.821 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89add9b640 10 monclient: dump: 2023-10-30T20:23:11.821 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:11.821 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.821 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.821 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.821 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:11.822 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.822 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:11.822 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:11.822 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:11.822 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89add9b640 10 monclient: _finish_auth 0 2023-10-30T20:23:11.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89add9b640 10 monclient: _check_auth_tickets 2023-10-30T20:23:11.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:11.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89add9b640 1 -- 172.21.15.183:0/290753097 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a00318c0 con 0x7f89b00b4440 2023-10-30T20:23:11.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89b4ba9640 5 monclient: authenticate success, global_id 4143 2023-10-30T20:23:11.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89b4ba9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:23:11.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89add9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:11.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89b4ba9640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:11.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89b4ba9640 4 set_mon_vals no callback set 2023-10-30T20:23:11.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89add9b640 1 -- 172.21.15.183:0/290753097 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a0031c60 con 0x7f89b00b4440 2023-10-30T20:23:11.825 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:11.825 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89add9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:23:11.825 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89add9b640 10 monclient: dump: 2023-10-30T20:23:11.825 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:11.825 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.826 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.826 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.826 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:11.826 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.826 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:11.827 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:11.827 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:11.827 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.827 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89b4ba9640 10 monclient: shutdown 2023-10-30T20:23:11.827 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89b4ba9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:11.828 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89b4ba9640 1 -- 172.21.15.183:0/290753097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b00b4440 msgr2=0x7f89b0144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:11.828 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89b4ba9640 1 --2- 172.21.15.183:0/290753097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b00b4440 0x7f89b0144ea0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f89b00b1780 tx=0x7f89a002b7f0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.828 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89b4ba9640 1 -- 172.21.15.183:0/290753097 shutdown_connections 2023-10-30T20:23:11.828 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89b4ba9640 1 --2- 172.21.15.183:0/290753097 >> v2:172.21.15.183:3300/0 conn(0x7f89b00b2230 0x7f89b014c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.828 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89b4ba9640 1 --2- 172.21.15.183:0/290753097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b00b4440 0x7f89b0144ea0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.828 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89b4ba9640 1 -- 172.21.15.183:0/290753097 >> 172.21.15.183:0/290753097 conn(0x7f89b0065ea0 msgr2=0x7f89b00662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:11.829 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89b4ba9640 1 -- 172.21.15.183:0/290753097 shutdown_connections 2023-10-30T20:23:11.829 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.813+0000 7f89b4ba9640 1 -- 172.21.15.183:0/290753097 wait complete. 2023-10-30T20:23:11.829 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.814+0000 7f89b4ba9640 10 monclient: build_initial_monmap 2023-10-30T20:23:11.829 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.814+0000 7f89b4ba9640 10 monclient: monmap: 2023-10-30T20:23:11.829 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2023-10-30T20:23:11.830 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:11.830 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2023-10-30T20:23:11.830 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2023-10-30T20:23:11.830 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:11.830 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2023-10-30T20:23:11.831 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:11.831 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:11.831 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:11.831 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.831 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.814+0000 7f89b4ba9640 1 librados: starting msgr at 2023-10-30T20:23:11.831 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.814+0000 7f89b4ba9640 1 librados: starting objecter 2023-10-30T20:23:11.832 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 1 Processor -- start 2023-10-30T20:23:11.832 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 1 -- start start 2023-10-30T20:23:11.832 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 1 librados: setting wanted keys 2023-10-30T20:23:11.832 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 1 librados: calling monclient init 2023-10-30T20:23:11.832 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 10 monclient: init 2023-10-30T20:23:11.833 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:11.833 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:23:11.833 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b00b2230 0x7f89b01da230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:11.833 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 10 monclient(hunting): picked mon.noname-a con 0x7f89b00b2230 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:11.833 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89b00b4440 0x7f89b013b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:11.834 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 10 monclient(hunting): picked mon.noname-b con 0x7f89b00b4440 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:11.834 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b01453e0 0x7f89b013b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:11.834 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 10 monclient(hunting): picked mon.noname-c con 0x7f89b01453e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:11.834 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:11.834 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b00a8d00 con 0x7f89b00b2230 2023-10-30T20:23:11.834 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:11.835 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b01cc100 con 0x7f89b00b4440 2023-10-30T20:23:11.835 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:11.835 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b01cc280 con 0x7f89b01453e0 2023-10-30T20:23:11.835 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:11.835 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.815+0000 7f89b4ba9640 10 monclient(hunting): authenticate will time out at 1511.017789s 2023-10-30T20:23:11.836 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.816+0000 7f89ae59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b00b2230 0x7f89b01da230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:11.836 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.816+0000 7f89aed9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b01453e0 0x7f89b013b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:11.836 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.816+0000 7f89ae59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b00b2230 0x7f89b01da230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.183:56008/0 (socket says 172.21.15.183:56008) 2023-10-30T20:23:11.836 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.816+0000 7f89ae59c640 1 -- 172.21.15.183:0/2156625092 learned_addr learned my addr 172.21.15.183:0/2156625092 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-10-30T20:23:11.836 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.816+0000 7f89ae59c640 10 monclient(hunting): get_auth_request con 0x7f89b00b2230 auth_method 0 2023-10-30T20:23:11.837 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.816+0000 7f89ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:11.837 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.816+0000 7f89ae59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:11.837 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.816+0000 7f89ae59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:11.837 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.816+0000 7f89aed9d640 10 monclient(hunting): get_auth_request con 0x7f89b01453e0 auth_method 0 2023-10-30T20:23:11.837 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.816+0000 7f89aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:11.838 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.816+0000 7f89aed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:11.838 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.816+0000 7f89aed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:11.838 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.817+0000 7f89ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:11.838 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.817+0000 7f89ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:11.838 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.817+0000 7f89ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:11.838 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.817+0000 7f89aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:11.839 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.817+0000 7f89aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:11.839 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.817+0000 7f89aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:11.839 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:11.818+0000 7f594b411040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T20:23:11.841 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.819+0000 7f89ae59c640 10 monclient(hunting): handle_auth_done global_id 4146 payload 824 2023-10-30T20:23:11.841 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.819+0000 7f89ae59c640 1 -- 172.21.15.183:0/2156625092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b01453e0 msgr2=0x7f89b013b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:11.841 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.819+0000 7f89ae59c640 1 --2- 172.21.15.183:0/2156625092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b01453e0 0x7f89b013b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.841 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.819+0000 7f89ae59c640 1 -- 172.21.15.183:0/2156625092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89b00b4440 msgr2=0x7f89b013b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:11.842 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.819+0000 7f89ae59c640 1 --2- 172.21.15.183:0/2156625092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89b00b4440 0x7f89b013b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:11.842 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.819+0000 7f89ae59c640 10 monclient: _finish_hunting 0 2023-10-30T20:23:11.842 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.819+0000 7f89ae59c640 1 monclient: found mon.noname-a 2023-10-30T20:23:11.842 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.819+0000 7f89ae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:11.842 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.819+0000 7f89ae59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:23:11.843 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.819+0000 7f89ae59c640 1 -- 172.21.15.183:0/2156625092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b01da770 con 0x7f89b00b2230 2023-10-30T20:23:11.843 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.822+0000 7f89ae59c640 1 --2- 172.21.15.183:0/2156625092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b00b2230 0x7f89b01da230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f89a0000c00 tx=0x7f89a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:11.843 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.822+0000 7f89a77fe640 10 client.?.objecter ms_handle_connect 0x7f89b00b2230 2023-10-30T20:23:11.843 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.822+0000 7f89a77fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:11.843 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89a77fe640 1 -- 172.21.15.183:0/2156625092 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a0033c80 con 0x7f89b00b2230 2023-10-30T20:23:11.844 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:11.844 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89a77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:23:11.844 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89a77fe640 10 monclient: dump: 2023-10-30T20:23:11.844 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:11.844 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.844 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.845 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.845 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:11.845 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.845 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:11.845 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:11.845 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:11.846 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.846 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89a77fe640 10 monclient: _finish_auth 0 2023-10-30T20:23:11.846 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89a77fe640 10 monclient: _check_auth_tickets 2023-10-30T20:23:11.846 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:11.846 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89a77fe640 1 -- 172.21.15.183:0/2156625092 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a0033df0 con 0x7f89b00b2230 2023-10-30T20:23:11.847 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89a77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:11.847 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89a77fe640 1 -- 172.21.15.183:0/2156625092 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a003d6b0 con 0x7f89b00b2230 2023-10-30T20:23:11.847 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:11.847 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:23:11.847 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89a77fe640 10 monclient: dump: 2023-10-30T20:23:11.848 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2023-10-30T20:23:11.848 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:11.848 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.848 INFO:teuthology.orchestra.run.smithi183.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:11.848 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:11.848 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2023-10-30T20:23:11.849 INFO:teuthology.orchestra.run.smithi183.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:11.849 INFO:teuthology.orchestra.run.smithi183.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:11.849 INFO:teuthology.orchestra.run.smithi183.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:11.849 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:23:11.849 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89b4ba9640 5 monclient: authenticate success, global_id 4146 2023-10-30T20:23:11.849 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89b4ba9640 10 monclient: _renew_subs 2023-10-30T20:23:11.850 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89ad59a640 4 set_mon_vals no callback set 2023-10-30T20:23:11.850 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89b4ba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:11.850 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89b4ba9640 1 -- 172.21.15.183:0/2156625092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b01da970 con 0x7f89b00b2230 2023-10-30T20:23:11.850 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89b4ba9640 10 client.4146.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:11.850 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89b4ba9640 10 monclient: _renew_subs 2023-10-30T20:23:11.851 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89b4ba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:11.851 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89b4ba9640 1 -- 172.21.15.183:0/2156625092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b0143d40 con 0x7f89b00b2230 2023-10-30T20:23:11.851 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.825+0000 7f89b4ba9640 1 librados: init done 2023-10-30T20:23:11.851 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.826+0000 7f89b4ba9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:11.851 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.826+0000 7f89b4ba9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:11.851 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.826+0000 7f89b4ba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:11.852 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.826+0000 7f89b4ba9640 1 -- 172.21.15.183:0/2156625092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b004e9b0 con 0x7f89b00b2230 2023-10-30T20:23:11.852 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.827+0000 7f89a77fe640 1 -- 172.21.15.183:0/2156625092 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f89a003a040 con 0x7f89b00b2230 2023-10-30T20:23:11.852 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.828+0000 7f89a77fe640 1 -- 172.21.15.183:0/2156625092 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f89a003db20 con 0x7f89b00b2230 2023-10-30T20:23:11.852 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.828+0000 7f89a77fe640 10 client.4146.objecter ms_dispatch 0x7f89b0000b90 osd_map(9..9 src has 1..9) v4 2023-10-30T20:23:11.852 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.828+0000 7f89a77fe640 3 client.4146.objecter handle_osd_map got epochs [9,9] > 0 2023-10-30T20:23:11.853 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.828+0000 7f89a77fe640 3 client.4146.objecter handle_osd_map decoding full epoch 9 2023-10-30T20:23:11.853 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.828+0000 7f89a77fe640 20 client.4146.objecter dump_active .. 0 homeless 2023-10-30T20:23:11.922 INFO:tasks.ceph.mgr.x.smithi183.stderr:2023-10-30T20:23:11.920+0000 7f594b411040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T20:23:11.926 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.924+0000 7f89a77fe640 1 -- 172.21.15.183:0/2156625092 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f89a002f050 con 0x7f89b00b2230 2023-10-30T20:23:11.926 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.925+0000 7f89a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:11.926 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.925+0000 7f89a77fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:11.994 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.993+0000 7f89b4ba9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d", "id": 7}] 2023-10-30T20:23:11.995 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.993+0000 7f89b4ba9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d", "id": 7}] 2023-10-30T20:23:11.995 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.993+0000 7f89b4ba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:11.995 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:11.993+0000 7f89b4ba9640 1 -- 172.21.15.183:0/2156625092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d", "id": 7} v 0) v1 -- 0x7f89b013d550 con 0x7f89b00b2230 2023-10-30T20:23:12.172 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.171+0000 7f89a77fe640 1 -- 172.21.15.183:0/2156625092 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 2) v1 ==== 44566+0+0 (secure 0 0 0) 0x7f89a003dcf0 con 0x7f89b00b2230 2023-10-30T20:23:12.412 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.411+0000 7f89a77fe640 1 -- 172.21.15.183:0/2156625092 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f89a003c590 con 0x7f89b00b2230 2023-10-30T20:23:12.412 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.411+0000 7f89a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d", "id": 7}] 2023-10-30T20:23:12.413 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.411+0000 7f89a77fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:12.413 INFO:teuthology.orchestra.run.smithi183.stdout:7 2023-10-30T20:23:12.414 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.412+0000 7f89b4ba9640 10 librados: watch_flush enter 2023-10-30T20:23:12.415 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.412+0000 7f89b4ba9640 10 librados: watch_flush exit 2023-10-30T20:23:12.415 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.412+0000 7f89b4ba9640 20 client.4146.objecter shutdown clearing up homeless session... 2023-10-30T20:23:12.415 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.412+0000 7f89b4ba9640 10 client.4146.objecter successfully canceled tick 2023-10-30T20:23:12.415 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.412+0000 7f89b4ba9640 10 monclient: shutdown 2023-10-30T20:23:12.415 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.412+0000 7f89b4ba9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:12.416 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.412+0000 7f89b4ba9640 1 -- 172.21.15.183:0/2156625092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b00b2230 msgr2=0x7f89b01da230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:12.416 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.412+0000 7f89b4ba9640 1 --2- 172.21.15.183:0/2156625092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b00b2230 0x7f89b01da230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f89a0000c00 tx=0x7f89a00046c0 comp rx=0 tx=0).stop 2023-10-30T20:23:12.416 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.412+0000 7f89b4ba9640 1 -- 172.21.15.183:0/2156625092 shutdown_connections 2023-10-30T20:23:12.416 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.412+0000 7f89b4ba9640 1 --2- 172.21.15.183:0/2156625092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b01453e0 0x7f89b013b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:12.416 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.412+0000 7f89b4ba9640 1 --2- 172.21.15.183:0/2156625092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89b00b4440 0x7f89b013b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:12.417 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.412+0000 7f89b4ba9640 1 --2- 172.21.15.183:0/2156625092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b00b2230 0x7f89b01da230 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:12.417 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.412+0000 7f89b4ba9640 1 -- 172.21.15.183:0/2156625092 >> 172.21.15.183:0/2156625092 conn(0x7f89b0065ea0 msgr2=0x7f89b00b78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:12.417 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.413+0000 7f89b4ba9640 1 -- 172.21.15.183:0/2156625092 shutdown_connections 2023-10-30T20:23:12.417 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.413+0000 7f89b4ba9640 1 -- 172.21.15.183:0/2156625092 wait complete. 2023-10-30T20:23:12.417 INFO:teuthology.orchestra.run.smithi183.stderr:2023-10-30T20:23:12.413+0000 7f89b4ba9640 1 librados: shutdown 2023-10-30T20:23:12.425 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T20:23:12.425 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-30T20:23:12.426 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-10-30T20:23:12.428 INFO:tasks.ceph.osd.0:Started 2023-10-30T20:23:12.428 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T20:23:12.429 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-30T20:23:12.429 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-10-30T20:23:12.431 INFO:tasks.ceph.osd.1:Started 2023-10-30T20:23:12.431 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T20:23:12.432 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-30T20:23:12.434 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-10-30T20:23:12.437 INFO:tasks.ceph.osd.2:Started 2023-10-30T20:23:12.437 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T20:23:12.437 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-30T20:23:12.438 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-10-30T20:23:12.440 INFO:tasks.ceph.osd.3:Started 2023-10-30T20:23:12.440 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T20:23:12.441 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-30T20:23:12.441 DEBUG:teuthology.orchestra.run.smithi183:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-10-30T20:23:12.444 INFO:tasks.ceph.osd.4:Started 2023-10-30T20:23:12.444 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T20:23:12.444 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-30T20:23:12.445 DEBUG:teuthology.orchestra.run.smithi183:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-10-30T20:23:12.447 INFO:tasks.ceph.osd.5:Started 2023-10-30T20:23:12.447 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T20:23:12.448 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-30T20:23:12.448 DEBUG:teuthology.orchestra.run.smithi183:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-10-30T20:23:12.450 INFO:tasks.ceph.osd.6:Started 2023-10-30T20:23:12.450 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T20:23:12.450 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-30T20:23:12.451 DEBUG:teuthology.orchestra.run.smithi183:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-10-30T20:23:12.453 INFO:tasks.ceph.osd.7:Started 2023-10-30T20:23:12.454 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T20:23:12.575 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.574+0000 7f10b9261640 10 monclient: get_monmap_and_config 2023-10-30T20:23:12.576 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.574+0000 7f10b9261640 10 monclient: build_initial_monmap 2023-10-30T20:23:12.576 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.574+0000 7f10b9261640 10 monclient: monmap: 2023-10-30T20:23:12.576 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:23:12.577 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:12.577 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:23:12.575040+0000 2023-10-30T20:23:12.577 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:23:12.575040+0000 2023-10-30T20:23:12.577 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:12.578 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:12.578 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:12.578 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:12.578 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:12.579 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:12.579 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:12.579 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:23:12.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.574+0000 7f10b9261640 1 Processor -- start 2023-10-30T20:23:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.574+0000 7f10b9261640 1 -- start start 2023-10-30T20:23:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.574+0000 7f10b9261640 10 monclient: init 2023-10-30T20:23:12.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.575+0000 7f10b9261640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:12.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.575+0000 7f10b9261640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-10-30T20:23:12.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.575+0000 7f10b9261640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b40b4390 0x7f10b40a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:12.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.575+0000 7f10b9261640 10 monclient(hunting): picked mon.noname-b con 0x7f10b40b4390 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:12.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.575+0000 7f10b9261640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10b41418a0 addr v1:172.21.15.183:6789/0 2023-10-30T20:23:12.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.575+0000 7f10b9261640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f10b4141ce0 0x7f10b40b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:12.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.575+0000 7f10b9261640 10 monclient(hunting): picked mon.noname-c con 0x7f10b4141ce0 addr v2:172.21.15.183:3300/0 2023-10-30T20:23:12.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.575+0000 7f10b9261640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b40a8f60 con 0x7f10b41418a0 2023-10-30T20:23:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.575+0000 7f10b9261640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.575+0000 7f10b9261640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b4067a10 con 0x7f10b4141ce0 2023-10-30T20:23:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.575+0000 7f10b9261640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.575+0000 7f10b9261640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b40b4770 con 0x7f10b40b4390 2023-10-30T20:23:12.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.575+0000 7f10b9261640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:12.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.575+0000 7f10b9261640 10 monclient(hunting): authenticate will time out at 1515.864509s 2023-10-30T20:23:12.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.576+0000 7f10b2ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b40b4390 0x7f10b40a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:12.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.576+0000 7f10b2ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b40b4390 0x7f10b40a8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46070/0 (socket says 172.21.15.69:46070) 2023-10-30T20:23:12.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.576+0000 7f10b2ffd640 1 -- 172.21.15.69:0/2531458544 learned_addr learned my addr 172.21.15.69:0/2531458544 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:23:12.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.576+0000 7f10b2ffd640 10 monclient(hunting): get_auth_request con 0x7f10b40b4390 auth_method 0 2023-10-30T20:23:12.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.576+0000 7f10b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:12.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.576+0000 7f10b2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:12.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.576+0000 7f10b2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:12.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.577+0000 7f10b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:12.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.577+0000 7f10b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:12.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.577+0000 7f10b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:12.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.578+0000 7f10b2ffd640 10 monclient(hunting): handle_auth_done global_id 4151 payload 274 2023-10-30T20:23:12.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.578+0000 7f10b2ffd640 1 -- 172.21.15.69:0/2531458544 >> v2:172.21.15.183:3300/0 conn(0x7f10b4141ce0 msgr2=0x7f10b40b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:12.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.578+0000 7f10b2ffd640 1 --2- 172.21.15.69:0/2531458544 >> v2:172.21.15.183:3300/0 conn(0x7f10b4141ce0 0x7f10b40b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:12.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.578+0000 7f10b2ffd640 1 -- 172.21.15.69:0/2531458544 >> v1:172.21.15.183:6789/0 conn(0x7f10b41418a0 legacy=0x7f10b40b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:12.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.578+0000 7f10b2ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:23:12.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.578+0000 7f10b2ffd640 1 monclient: found mon.noname-b 2023-10-30T20:23:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.578+0000 7f10b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.578+0000 7f10b2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:23:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.578+0000 7f10b2ffd640 1 -- 172.21.15.69:0/2531458544 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b4142170 con 0x7f10b40b4390 2023-10-30T20:23:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.580+0000 7f10b2ffd640 1 --2- 172.21.15.69:0/2531458544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b40b4390 0x7f10b40a8770 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009870 tx=0x7f10a402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9149d0a4451cdda0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:12.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b1ffb640 1 -- 172.21.15.69:0/2531458544 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4031640 con 0x7f10b40b4390 2023-10-30T20:23:12.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:12.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:23:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b1ffb640 10 monclient: dump: 2023-10-30T20:23:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:23:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:12.597 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:12.597 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:12.597 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:12.597 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:12.597 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:12.598 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:12.598 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:23:12.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b1ffb640 10 monclient: _finish_auth 0 2023-10-30T20:23:12.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b1ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:23:12.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:12.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b1ffb640 1 -- 172.21.15.69:0/2531458544 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a40317b0 con 0x7f10b40b4390 2023-10-30T20:23:12.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:12.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b1ffb640 1 -- 172.21.15.69:0/2531458544 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4031b50 con 0x7f10b40b4390 2023-10-30T20:23:12.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:12.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:23:12.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b1ffb640 10 monclient: dump: 2023-10-30T20:23:12.600 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:23:12.600 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:12.600 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:12.601 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:12.601 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:12.601 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:12.601 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:12.601 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:12.601 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:12.602 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:23:12.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b9261640 5 monclient: authenticate success, global_id 4151 2023-10-30T20:23:12.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b9261640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:12.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.584+0000 7f10b9261640 4 set_mon_vals no callback set 2023-10-30T20:23:12.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.585+0000 7f10b9261640 10 monclient: shutdown 2023-10-30T20:23:12.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.585+0000 7f10b9261640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:12.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.585+0000 7f10b9261640 1 -- 172.21.15.69:0/2531458544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b40b4390 msgr2=0x7f10b40a8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:12.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.585+0000 7f10b9261640 1 --2- 172.21.15.69:0/2531458544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b40b4390 0x7f10b40a8770 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009870 tx=0x7f10a402b8f0 comp rx=0 tx=0).stop 2023-10-30T20:23:12.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.585+0000 7f10b9261640 1 -- 172.21.15.69:0/2531458544 shutdown_connections 2023-10-30T20:23:12.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.585+0000 7f10b9261640 1 --2- 172.21.15.69:0/2531458544 >> v2:172.21.15.183:3300/0 conn(0x7f10b4141ce0 0x7f10b40b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:12.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.585+0000 7f10b9261640 1 --2- 172.21.15.69:0/2531458544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b40b4390 0x7f10b40a8770 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:12.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.585+0000 7f10b9261640 1 -- 172.21.15.69:0/2531458544 >> 172.21.15.69:0/2531458544 conn(0x7f10b4065ea0 msgr2=0x7f10b40662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:12.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.585+0000 7f10b9261640 1 -- 172.21.15.69:0/2531458544 shutdown_connections 2023-10-30T20:23:12.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.585+0000 7f10b9261640 1 -- 172.21.15.69:0/2531458544 wait complete. 2023-10-30T20:23:12.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.586+0000 7f10b9261640 10 monclient: build_initial_monmap 2023-10-30T20:23:12.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.586+0000 7f10b9261640 10 monclient: monmap: 2023-10-30T20:23:12.606 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:23:12.606 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:12.606 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:23:12.606 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:23:12.606 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:23:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:12.607 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:23:12.608 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.586+0000 7f10b9261640 1 librados: starting msgr at 2023-10-30T20:23:12.608 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.586+0000 7f10b9261640 1 librados: starting objecter 2023-10-30T20:23:12.608 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.587+0000 7f10b9261640 1 Processor -- start 2023-10-30T20:23:12.608 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.587+0000 7f10b9261640 1 -- start start 2023-10-30T20:23:12.608 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.587+0000 7f10b9261640 1 librados: setting wanted keys 2023-10-30T20:23:12.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.587+0000 7f10b9261640 1 librados: calling monclient init 2023-10-30T20:23:12.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.587+0000 7f10b9261640 10 monclient: init 2023-10-30T20:23:12.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.588+0000 7f10b9261640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:12.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.588+0000 7f10b9261640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:23:12.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.588+0000 7f10b9261640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10b40b4390 0x7f10b41da4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.588+0000 7f10b9261640 10 monclient(hunting): picked mon.noname-b con 0x7f10b40b4390 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.588+0000 7f10b9261640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b41418a0 0x7f10b4144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.588+0000 7f10b9261640 10 monclient(hunting): picked mon.noname-c con 0x7f10b41418a0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.588+0000 7f10b9261640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10b4141ce0 0x7f10b41447d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b9261640 10 monclient(hunting): picked mon.noname-a con 0x7f10b4141ce0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b9261640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b9261640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b40b12b0 con 0x7f10b4141ce0 2023-10-30T20:23:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b9261640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b9261640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b4143a90 con 0x7f10b40b4390 2023-10-30T20:23:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b9261640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b9261640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b4143e00 con 0x7f10b41418a0 2023-10-30T20:23:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b9261640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b9261640 10 monclient(hunting): authenticate will time out at 1515.877973s 2023-10-30T20:23:12.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b27fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b41418a0 0x7f10b4144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:12.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b27fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b41418a0 0x7f10b4144290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46084/0 (socket says 172.21.15.69:46084) 2023-10-30T20:23:12.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b27fc640 1 -- 172.21.15.69:0/2638951967 learned_addr learned my addr 172.21.15.69:0/2638951967 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:23:12.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b27fc640 10 monclient(hunting): get_auth_request con 0x7f10b41418a0 auth_method 0 2023-10-30T20:23:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.589+0000 7f10b27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.590+0000 7f10b27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:12.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.590+0000 7f10b27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:12.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.590+0000 7f10b27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:12.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.594+0000 7f10b27fc640 10 monclient(hunting): handle_auth_done global_id 4154 payload 824 2023-10-30T20:23:12.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.594+0000 7f10b27fc640 1 -- 172.21.15.69:0/2638951967 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10b40b4390 msgr2=0x7f10b41da4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:12.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.594+0000 7f10b27fc640 1 --2- 172.21.15.69:0/2638951967 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10b40b4390 0x7f10b41da4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:12.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.594+0000 7f10b27fc640 1 -- 172.21.15.69:0/2638951967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10b4141ce0 msgr2=0x7f10b41447d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:12.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.594+0000 7f10b27fc640 1 --2- 172.21.15.69:0/2638951967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10b4141ce0 0x7f10b41447d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:12.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.594+0000 7f10b27fc640 10 monclient: _finish_hunting 0 2023-10-30T20:23:12.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.594+0000 7f10b27fc640 1 monclient: found mon.noname-c 2023-10-30T20:23:12.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.594+0000 7f10b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:12.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.594+0000 7f10b27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:23:12.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.594+0000 7f10b27fc640 1 -- 172.21.15.69:0/2638951967 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b4142d70 con 0x7f10b41418a0 2023-10-30T20:23:12.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.598+0000 7f10b27fc640 1 --2- 172.21.15.69:0/2638951967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b41418a0 0x7f10b4144290 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f10a800c440 tx=0x7f10a800c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:12.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.598+0000 7f109b7fe640 10 client.?.objecter ms_handle_connect 0x7f10b41418a0 2023-10-30T20:23:12.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.598+0000 7f109b7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:12.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f109b7fe640 1 -- 172.21.15.69:0/2638951967 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a80035f0 con 0x7f10b41418a0 2023-10-30T20:23:12.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f109b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:12.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f109b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:23:12.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f109b7fe640 10 monclient: dump: 2023-10-30T20:23:12.622 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:23:12.623 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:12.623 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:12.623 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:12.623 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:12.623 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:12.624 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:12.624 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:12.624 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:12.624 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:23:12.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f109b7fe640 10 monclient: _finish_auth 0 2023-10-30T20:23:12.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f109b7fe640 10 monclient: _check_auth_tickets 2023-10-30T20:23:12.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f109b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:12.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f109b7fe640 1 -- 172.21.15.69:0/2638951967 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a800e040 con 0x7f10b41418a0 2023-10-30T20:23:12.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f109b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:12.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f109b7fe640 1 -- 172.21.15.69:0/2638951967 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a80039c0 con 0x7f10b41418a0 2023-10-30T20:23:12.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f109b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:12.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f109b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:23:12.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f109b7fe640 10 monclient: dump: 2023-10-30T20:23:12.627 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:23:12.627 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:12.627 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:12.627 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:12.627 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:12.628 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:12.628 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:12.628 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:12.628 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:12.628 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:23:12.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f10b17fa640 4 set_mon_vals no callback set 2023-10-30T20:23:12.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f10b9261640 5 monclient: authenticate success, global_id 4154 2023-10-30T20:23:12.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f10b9261640 10 monclient: _renew_subs 2023-10-30T20:23:12.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f10b9261640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:12.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.603+0000 7f10b9261640 1 -- 172.21.15.69:0/2638951967 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b4144da0 con 0x7f10b41418a0 2023-10-30T20:23:12.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.604+0000 7f10b9261640 10 client.4154.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:12.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.604+0000 7f10b9261640 10 monclient: _renew_subs 2023-10-30T20:23:12.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.604+0000 7f10b9261640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:12.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.604+0000 7f10b9261640 1 -- 172.21.15.69:0/2638951967 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b40b5590 con 0x7f10b41418a0 2023-10-30T20:23:12.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.604+0000 7f10b9261640 1 librados: init done 2023-10-30T20:23:12.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.605+0000 7f10b9261640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:12.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.606+0000 7f10b9261640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:12.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.606+0000 7f10b9261640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:12.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.606+0000 7f10b9261640 1 -- 172.21.15.69:0/2638951967 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b404e9b0 con 0x7f10b41418a0 2023-10-30T20:23:12.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.623+0000 7f109b7fe640 1 -- 172.21.15.69:0/2638951967 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 2) v1 ==== 44566+0+0 (secure 0 0 0) 0x7f10a8003b30 con 0x7f10b41418a0 2023-10-30T20:23:12.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.625+0000 7f109b7fe640 1 -- 172.21.15.69:0/2638951967 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f10a8003de0 con 0x7f10b41418a0 2023-10-30T20:23:12.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.625+0000 7f109b7fe640 10 client.4154.objecter ms_dispatch 0x7f10b4000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T20:23:12.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.625+0000 7f109b7fe640 3 client.4154.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T20:23:12.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.625+0000 7f109b7fe640 3 client.4154.objecter handle_osd_map decoding full epoch 10 2023-10-30T20:23:12.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.625+0000 7f109b7fe640 20 client.4154.objecter dump_active .. 0 homeless 2023-10-30T20:23:12.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.791+0000 7f109b7fe640 1 -- 172.21.15.69:0/2638951967 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f10a80104c0 con 0x7f10b41418a0 2023-10-30T20:23:12.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.791+0000 7f109b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:12.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.791+0000 7f109b7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:12.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.913+0000 7f10b9261640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:12.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.913+0000 7f10b9261640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:12.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.913+0000 7f10b9261640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:12.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:12.913+0000 7f10b9261640 1 -- 172.21.15.69:0/2638951967 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10b4142d70 con 0x7f10b41418a0 2023-10-30T20:23:13.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.123+0000 7f109b7fe640 1 -- 172.21.15.69:0/2638951967 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f10a8011ab0 con 0x7f10b41418a0 2023-10-30T20:23:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.123+0000 7f109b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.123+0000 7f109b7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:13.126 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:23:13.126 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":10,"fsid":"32c1a723-7694-47d8-8b37-b1257dfe1796","created":"2023-10-30T20:23:06.559937+0000","modified":"2023-10-30T20:23:12.023084+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T20:23:12.023084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af10bc15-318a-4349-af76-32a30c8acf80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d0d32a9a-e24a-4d0c-9b90-125485d70672","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"508f2c3a-9e54-46aa-b6c0-52fad97fd341","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"003a2613-1050-4a96-b927-0e5983687319","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"678fc58e-ed6a-4fd0-9825-93297e3af6f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f945159a-a53c-48ba-97df-9374f9e56ffd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8184f59d-15e2-4bdc-90e3-fe1e231ca125","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T20:23:13.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.124+0000 7f10b9261640 10 librados: watch_flush enter 2023-10-30T20:23:13.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.124+0000 7f10b9261640 10 librados: watch_flush exit 2023-10-30T20:23:13.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.124+0000 7f10b9261640 20 client.4154.objecter shutdown clearing up homeless session... 2023-10-30T20:23:13.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.124+0000 7f10b9261640 10 client.4154.objecter successfully canceled tick 2023-10-30T20:23:13.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.125+0000 7f10b9261640 10 monclient: shutdown 2023-10-30T20:23:13.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.125+0000 7f10b9261640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:13.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.125+0000 7f10b9261640 1 -- 172.21.15.69:0/2638951967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b41418a0 msgr2=0x7f10b4144290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:13.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.125+0000 7f10b9261640 1 --2- 172.21.15.69:0/2638951967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b41418a0 0x7f10b4144290 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f10a800c440 tx=0x7f10a800c900 comp rx=0 tx=0).stop 2023-10-30T20:23:13.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.125+0000 7f10b9261640 1 -- 172.21.15.69:0/2638951967 shutdown_connections 2023-10-30T20:23:13.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.125+0000 7f10b9261640 1 --2- 172.21.15.69:0/2638951967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10b4141ce0 0x7f10b41447d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:13.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.125+0000 7f10b9261640 1 --2- 172.21.15.69:0/2638951967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b41418a0 0x7f10b4144290 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:13.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.125+0000 7f10b9261640 1 --2- 172.21.15.69:0/2638951967 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10b40b4390 0x7f10b41da4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:13.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.125+0000 7f10b9261640 1 -- 172.21.15.69:0/2638951967 >> 172.21.15.69:0/2638951967 conn(0x7f10b4065ea0 msgr2=0x7f10b413bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:13.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.125+0000 7f10b9261640 1 -- 172.21.15.69:0/2638951967 shutdown_connections 2023-10-30T20:23:13.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.125+0000 7f10b9261640 1 -- 172.21.15.69:0/2638951967 wait complete. 2023-10-30T20:23:13.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:23:13.125+0000 7f10b9261640 1 librados: shutdown 2023-10-30T20:23:13.147 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-30T20:23:13.148 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-30T20:23:13.833 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:13.832+0000 7fea126d0640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-30T20:23:13.912 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:23:13.911+0000 7fea126d0640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-30T20:23:13.953 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T20:23:14.102 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.100+0000 7ff849456640 10 monclient: get_monmap_and_config 2023-10-30T20:23:14.103 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.100+0000 7ff849456640 10 monclient: build_initial_monmap 2023-10-30T20:23:14.103 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.100+0000 7ff849456640 10 monclient: monmap: 2023-10-30T20:23:14.103 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-10-30T20:23:14.103 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:14.104 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:23:14.101864+0000 2023-10-30T20:23:14.104 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:23:14.101864+0000 2023-10-30T20:23:14.104 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:14.104 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:14.104 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:14.105 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:14.105 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:14.105 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:14.105 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:14.105 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:14.106 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.101+0000 7ff849456640 1 Processor -- start 2023-10-30T20:23:14.106 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.102+0000 7ff849456640 1 -- start start 2023-10-30T20:23:14.106 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.102+0000 7ff849456640 10 monclient: init 2023-10-30T20:23:14.107 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.102+0000 7ff849456640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:14.108 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.103+0000 7ff849456640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-10-30T20:23:14.108 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.103+0000 7ff849456640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8440b2db0 0x7ff8440afa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:14.108 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.103+0000 7ff849456640 10 monclient(hunting): picked mon.noname-a con 0x7ff8440b2db0 addr v2:172.21.15.69:3300/0 2023-10-30T20:23:14.108 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.103+0000 7ff849456640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8440b0630 addr v1:172.21.15.69:6789/0 2023-10-30T20:23:14.108 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.103+0000 7ff849456640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff844070cd0 0x7ff844071120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:14.109 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.103+0000 7ff849456640 10 monclient(hunting): picked mon.noname-c con 0x7ff844070cd0 addr v2:172.21.15.183:3300/0 2023-10-30T20:23:14.109 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.103+0000 7ff849456640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84406c5b0 con 0x7ff8440b0630 2023-10-30T20:23:14.109 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.103+0000 7ff849456640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:14.109 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.103+0000 7ff849456640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff844067a10 con 0x7ff8440b2db0 2023-10-30T20:23:14.109 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.103+0000 7ff849456640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:14.110 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.103+0000 7ff849456640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8440b3190 con 0x7ff844070cd0 2023-10-30T20:23:14.110 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.103+0000 7ff849456640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:14.110 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.103+0000 7ff849456640 10 monclient(hunting): authenticate will time out at 1517.391953s 2023-10-30T20:23:14.126 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.124+0000 7ff842ffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8440b2db0 0x7ff8440afa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:14.126 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.124+0000 7ff8427fc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff8440b0630 0x7ff844145e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53914/0 (socket says 172.21.15.69:53914) 2023-10-30T20:23:14.127 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.124+0000 7ff8427fc640 1 -- 172.21.15.69:0/1048855433 learned_addr learned my addr 172.21.15.69:0/1048855433 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:23:14.142 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.141+0000 7ff8437fe640 1 --2- 172.21.15.69:0/1048855433 >> v2:172.21.15.183:3300/0 conn(0x7ff844070cd0 0x7ff844071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:14.157 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.156+0000 7ff842ffd640 10 monclient(hunting): get_auth_request con 0x7ff8440b2db0 auth_method 0 2023-10-30T20:23:14.157 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.156+0000 7ff842ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:14.158 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.156+0000 7ff842ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:14.158 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.156+0000 7ff842ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:14.173 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.171+0000 7ff8437fe640 10 monclient(hunting): get_auth_request con 0x7ff844070cd0 auth_method 0 2023-10-30T20:23:14.173 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.171+0000 7ff8437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:14.173 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.171+0000 7ff8437fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:14.174 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.171+0000 7ff8437fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:14.174 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.173+0000 7ff842ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:14.175 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.173+0000 7ff842ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:14.175 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.173+0000 7ff842ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:14.178 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.177+0000 7ff8437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:14.179 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.177+0000 7ff8437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:14.179 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.177+0000 7ff8437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:14.180 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.178+0000 7ff842ffd640 10 monclient(hunting): handle_auth_done global_id 4164 payload 274 2023-10-30T20:23:14.180 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.178+0000 7ff842ffd640 1 -- 172.21.15.69:0/1048855433 >> v2:172.21.15.183:3300/0 conn(0x7ff844070cd0 msgr2=0x7ff844071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:14.180 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.178+0000 7ff842ffd640 1 --2- 172.21.15.69:0/1048855433 >> v2:172.21.15.183:3300/0 conn(0x7ff844070cd0 0x7ff844071120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:14.180 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.178+0000 7ff842ffd640 1 -- 172.21.15.69:0/1048855433 >> v1:172.21.15.69:6789/0 conn(0x7ff8440b0630 legacy=0x7ff844145e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:14.181 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.178+0000 7ff842ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:23:14.181 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.178+0000 7ff842ffd640 1 monclient: found mon.noname-a 2023-10-30T20:23:14.181 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.178+0000 7ff842ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:14.181 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.178+0000 7ff842ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:23:14.181 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.178+0000 7ff842ffd640 1 -- 172.21.15.69:0/1048855433 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8440af5a0 con 0x7ff8440b2db0 2023-10-30T20:23:14.191 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.190+0000 7ff842ffd640 1 --2- 172.21.15.69:0/1048855433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8440b2db0 0x7ff8440afa80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff84406c9f0 tx=0x7ff83802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d7a086737c93c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:14.234 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff841ffb640 1 -- 172.21.15.69:0/1048855433 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8380315a0 con 0x7ff8440b2db0 2023-10-30T20:23:14.234 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff841ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:14.234 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff841ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:23:14.234 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff841ffb640 10 monclient: dump: 2023-10-30T20:23:14.235 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:14.235 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:14.235 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:14.235 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:14.235 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:14.235 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:14.236 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:14.236 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:14.236 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:14.236 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:14.236 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff841ffb640 10 monclient: _finish_auth 0 2023-10-30T20:23:14.237 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff841ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:23:14.237 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff841ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:14.237 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff841ffb640 1 -- 172.21.15.69:0/1048855433 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff838031710 con 0x7ff8440b2db0 2023-10-30T20:23:14.237 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff849456640 5 monclient: authenticate success, global_id 4164 2023-10-30T20:23:14.237 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff849456640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:23:14.238 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff841ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:14.238 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff849456640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:14.238 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff841ffb640 1 -- 172.21.15.69:0/1048855433 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff838031ab0 con 0x7ff8440b2db0 2023-10-30T20:23:14.238 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff841ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:14.239 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff849456640 4 set_mon_vals no callback set 2023-10-30T20:23:14.239 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff841ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:23:14.239 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.232+0000 7ff841ffb640 10 monclient: dump: 2023-10-30T20:23:14.239 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:14.239 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:14.239 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:14.240 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:14.240 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:14.240 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:14.240 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:14.240 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:14.240 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:14.241 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:14.246 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.233+0000 7ff849456640 10 monclient: shutdown 2023-10-30T20:23:14.246 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.233+0000 7ff849456640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:14.247 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.233+0000 7ff849456640 1 -- 172.21.15.69:0/1048855433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8440b2db0 msgr2=0x7ff8440afa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:14.247 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.233+0000 7ff849456640 1 --2- 172.21.15.69:0/1048855433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8440b2db0 0x7ff8440afa80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff84406c9f0 tx=0x7ff83802b840 comp rx=0 tx=0).stop 2023-10-30T20:23:14.247 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.234+0000 7ff849456640 1 -- 172.21.15.69:0/1048855433 shutdown_connections 2023-10-30T20:23:14.247 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.234+0000 7ff849456640 1 --2- 172.21.15.69:0/1048855433 >> v2:172.21.15.183:3300/0 conn(0x7ff844070cd0 0x7ff844071120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:14.247 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.234+0000 7ff849456640 1 --2- 172.21.15.69:0/1048855433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8440b2db0 0x7ff8440afa80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:14.248 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.234+0000 7ff849456640 1 -- 172.21.15.69:0/1048855433 >> 172.21.15.69:0/1048855433 conn(0x7ff8440b3e10 msgr2=0x7ff8440b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:14.248 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.234+0000 7ff849456640 1 -- 172.21.15.69:0/1048855433 shutdown_connections 2023-10-30T20:23:14.248 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.234+0000 7ff849456640 1 -- 172.21.15.69:0/1048855433 wait complete. 2023-10-30T20:23:14.248 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.235+0000 7ff849456640 10 monclient: build_initial_monmap 2023-10-30T20:23:14.248 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.235+0000 7ff849456640 10 monclient: monmap: 2023-10-30T20:23:14.248 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-10-30T20:23:14.249 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:14.249 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2023-10-30T20:23:14.249 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2023-10-30T20:23:14.249 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:14.249 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-10-30T20:23:14.250 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:14.250 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:14.250 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:14.250 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:14.250 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.235+0000 7ff849456640 1 librados: starting msgr at 2023-10-30T20:23:14.251 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.235+0000 7ff849456640 1 librados: starting objecter 2023-10-30T20:23:14.251 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.236+0000 7ff849456640 1 Processor -- start 2023-10-30T20:23:14.251 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.236+0000 7ff849456640 1 -- start start 2023-10-30T20:23:14.251 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.236+0000 7ff849456640 1 librados: setting wanted keys 2023-10-30T20:23:14.251 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.236+0000 7ff849456640 1 librados: calling monclient init 2023-10-30T20:23:14.252 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.236+0000 7ff849456640 10 monclient: init 2023-10-30T20:23:14.252 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:14.252 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:23:14.252 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff844070cd0 0x7ff8441cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:14.252 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 10 monclient(hunting): picked mon.noname-b con 0x7ff844070cd0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:14.253 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8440b0630 0x7ff8441dddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:14.253 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 10 monclient(hunting): picked mon.noname-c con 0x7ff8440b0630 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:14.253 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8440b2db0 0x7ff84406d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:14.253 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 10 monclient(hunting): picked mon.noname-a con 0x7ff8440b2db0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:14.253 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:14.253 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8440adb70 con 0x7ff8440b2db0 2023-10-30T20:23:14.254 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:14.254 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84406cbe0 con 0x7ff844070cd0 2023-10-30T20:23:14.254 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:14.254 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8441453d0 con 0x7ff8440b0630 2023-10-30T20:23:14.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:14.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.237+0000 7ff849456640 10 monclient(hunting): authenticate will time out at 1517.526662s 2023-10-30T20:23:14.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.238+0000 7ff8427fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8440b0630 0x7ff8441dddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:14.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.238+0000 7ff8427fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8440b0630 0x7ff8441dddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46090/0 (socket says 172.21.15.69:46090) 2023-10-30T20:23:14.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.238+0000 7ff8427fc640 1 -- 172.21.15.69:0/3982173759 learned_addr learned my addr 172.21.15.69:0/3982173759 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:23:14.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.238+0000 7ff8427fc640 10 monclient(hunting): get_auth_request con 0x7ff8440b0630 auth_method 0 2023-10-30T20:23:14.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.238+0000 7ff8427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:14.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.238+0000 7ff8427fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:14.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.238+0000 7ff8427fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:14.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.238+0000 7ff842ffd640 1 --2- 172.21.15.69:0/3982173759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff844070cd0 0x7ff8441cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:14.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.239+0000 7ff842ffd640 10 monclient(hunting): get_auth_request con 0x7ff844070cd0 auth_method 0 2023-10-30T20:23:14.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.239+0000 7ff842ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:14.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.239+0000 7ff842ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:14.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.239+0000 7ff842ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:14.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.239+0000 7ff8427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:14.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.239+0000 7ff8427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:14.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.239+0000 7ff8427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:14.258 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.239+0000 7ff842ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:14.258 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.239+0000 7ff842ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:14.258 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.239+0000 7ff842ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:14.258 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.243+0000 7ff8427fc640 10 monclient(hunting): handle_auth_done global_id 4157 payload 824 2023-10-30T20:23:14.258 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.243+0000 7ff8427fc640 1 -- 172.21.15.69:0/3982173759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff844070cd0 msgr2=0x7ff8441cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:14.259 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.243+0000 7ff8427fc640 1 --2- 172.21.15.69:0/3982173759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff844070cd0 0x7ff8441cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:14.259 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.243+0000 7ff8427fc640 1 -- 172.21.15.69:0/3982173759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8440b2db0 msgr2=0x7ff84406d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:14.259 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.243+0000 7ff8427fc640 1 --2- 172.21.15.69:0/3982173759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8440b2db0 0x7ff84406d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:14.259 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.243+0000 7ff8427fc640 10 monclient: _finish_hunting 0 2023-10-30T20:23:14.260 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.243+0000 7ff8427fc640 1 monclient: found mon.noname-c 2023-10-30T20:23:14.260 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.243+0000 7ff8427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:14.260 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.243+0000 7ff8427fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:23:14.260 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.243+0000 7ff8427fc640 1 -- 172.21.15.69:0/3982173759 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84406da60 con 0x7ff8440b0630 2023-10-30T20:23:14.261 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.247+0000 7ff8427fc640 1 --2- 172.21.15.69:0/3982173759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8440b0630 0x7ff8441dddd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff83400c4a0 tx=0x7ff83400c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:14.261 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.247+0000 7ff82b7fe640 10 client.?.objecter ms_handle_connect 0x7ff8440b0630 2023-10-30T20:23:14.262 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.247+0000 7ff82b7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:14.262 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff82b7fe640 1 -- 172.21.15.69:0/3982173759 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff834005ce0 con 0x7ff8440b0630 2023-10-30T20:23:14.262 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff82b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:14.262 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff82b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:23:14.262 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff82b7fe640 10 monclient: dump: 2023-10-30T20:23:14.263 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:14.263 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:14.263 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:14.263 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:14.263 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:14.264 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:14.264 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:14.264 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:14.264 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:14.264 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:14.264 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff82b7fe640 10 monclient: _finish_auth 0 2023-10-30T20:23:14.265 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff82b7fe640 10 monclient: _check_auth_tickets 2023-10-30T20:23:14.265 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff82b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:14.265 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff849456640 5 monclient: authenticate success, global_id 4157 2023-10-30T20:23:14.265 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff849456640 10 monclient: _renew_subs 2023-10-30T20:23:14.265 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff849456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:14.265 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff849456640 1 -- 172.21.15.69:0/3982173759 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff844146010 con 0x7ff8440b0630 2023-10-30T20:23:14.266 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff849456640 10 client.4157.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:14.266 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff849456640 10 monclient: _renew_subs 2023-10-30T20:23:14.266 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff849456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:14.266 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff849456640 1 -- 172.21.15.69:0/3982173759 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff84406d820 con 0x7ff8440b0630 2023-10-30T20:23:14.266 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.250+0000 7ff849456640 1 librados: init done 2023-10-30T20:23:14.267 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.252+0000 7ff849456640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:14.267 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.252+0000 7ff849456640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:14.267 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.252+0000 7ff849456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:14.267 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.252+0000 7ff849456640 1 -- 172.21.15.69:0/3982173759 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84406ca40 con 0x7ff8440b0630 2023-10-30T20:23:14.694 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.693+0000 7ff82b7fe640 1 -- 172.21.15.69:0/3982173759 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83400e040 con 0x7ff8440b0630 2023-10-30T20:23:14.694 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.693+0000 7ff82b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:14.695 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.693+0000 7ff82b7fe640 1 -- 172.21.15.69:0/3982173759 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8340072d0 con 0x7ff8440b0630 2023-10-30T20:23:14.695 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.693+0000 7ff82b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:14.695 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.693+0000 7ff82b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:23:14.695 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.693+0000 7ff82b7fe640 10 monclient: dump: 2023-10-30T20:23:14.696 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:14.696 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:14.696 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:14.696 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:14.697 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:14.697 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:14.697 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:14.697 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:14.697 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:14.698 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:14.698 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.693+0000 7ff8417fa640 4 set_mon_vals no callback set 2023-10-30T20:23:14.699 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.694+0000 7ff82b7fe640 1 -- 172.21.15.69:0/3982173759 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 3) v1 ==== 89032+0+0 (secure 0 0 0) 0x7ff834007500 con 0x7ff8440b0630 2023-10-30T20:23:14.699 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.694+0000 7ff82b7fe640 1 --2- 172.21.15.69:0/3982173759 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff83006c610 0x7ff83006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:14.699 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.695+0000 7ff842ffd640 1 --2- 172.21.15.69:0/3982173759 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff83006c610 0x7ff83006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:14.699 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.695+0000 7ff842ffd640 10 monclient: get_auth_request con 0x7ff83006c610 auth_method 0 2023-10-30T20:23:14.699 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.695+0000 7ff842ffd640 1 --2- 172.21.15.69:0/3982173759 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff83006c610 0x7ff83006ead0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff838036000 tx=0x7ff83803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:14.700 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.695+0000 7ff82b7fe640 10 client.4157.objecter ms_handle_connect 0x7ff83006c610 2023-10-30T20:23:14.716 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.714+0000 7ff82b7fe640 1 -- 172.21.15.69:0/3982173759 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7ff834057920 con 0x7ff8440b0630 2023-10-30T20:23:14.716 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.714+0000 7ff82b7fe640 10 client.4157.objecter ms_dispatch 0x7ff844000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T20:23:14.716 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.714+0000 7ff82b7fe640 3 client.4157.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T20:23:14.716 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.714+0000 7ff82b7fe640 3 client.4157.objecter handle_osd_map decoding full epoch 10 2023-10-30T20:23:14.717 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.714+0000 7ff82b7fe640 20 client.4157.objecter dump_active .. 0 homeless 2023-10-30T20:23:14.717 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.714+0000 7ff82b7fe640 1 -- 172.21.15.69:0/3982173759 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff83405b9c0 con 0x7ff8440b0630 2023-10-30T20:23:14.717 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.714+0000 7ff82b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:14.717 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:14.714+0000 7ff82b7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:15.074 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.073+0000 7ff849456640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:15.075 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.073+0000 7ff849456640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:15.075 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.073+0000 7ff849456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:15.075 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.073+0000 7ff849456640 1 -- 172.21.15.69:0/3982173759 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff84406ce20 con 0x7ff8440b0630 2023-10-30T20:23:15.106 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.105+0000 7ff82b7fe640 1 -- 172.21.15.69:0/3982173759 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7ff834053710 con 0x7ff8440b0630 2023-10-30T20:23:15.298 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.297+0000 7ff82b7fe640 1 -- 172.21.15.69:0/3982173759 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7ff83405bc20 con 0x7ff8440b0630 2023-10-30T20:23:15.299 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.297+0000 7ff82b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:15.299 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.297+0000 7ff82b7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:15.299 INFO:teuthology.misc.health.smithi069.stdout: 2023-10-30T20:23:15.299 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"32c1a723-7694-47d8-8b37-b1257dfe1796","created":"2023-10-30T20:23:06.559937+0000","modified":"2023-10-30T20:23:12.023084+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T20:23:12.023084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af10bc15-318a-4349-af76-32a30c8acf80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d0d32a9a-e24a-4d0c-9b90-125485d70672","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"508f2c3a-9e54-46aa-b6c0-52fad97fd341","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"003a2613-1050-4a96-b927-0e5983687319","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"678fc58e-ed6a-4fd0-9825-93297e3af6f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f945159a-a53c-48ba-97df-9374f9e56ffd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8184f59d-15e2-4bdc-90e3-fe1e231ca125","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T20:23:15.302 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 10 librados: watch_flush enter 2023-10-30T20:23:15.302 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 10 librados: watch_flush exit 2023-10-30T20:23:15.302 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 20 client.4157.objecter shutdown clearing up homeless session... 2023-10-30T20:23:15.302 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 10 client.4157.objecter successfully canceled tick 2023-10-30T20:23:15.303 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 1 -- 172.21.15.69:0/3982173759 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff83006c610 msgr2=0x7ff83006ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:15.303 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 1 --2- 172.21.15.69:0/3982173759 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff83006c610 0x7ff83006ead0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff838036000 tx=0x7ff83803d040 comp rx=0 tx=0).stop 2023-10-30T20:23:15.303 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 10 monclient: shutdown 2023-10-30T20:23:15.303 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:15.303 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 1 -- 172.21.15.69:0/3982173759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8440b0630 msgr2=0x7ff8441dddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:15.303 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 1 --2- 172.21.15.69:0/3982173759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8440b0630 0x7ff8441dddd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff83400c4a0 tx=0x7ff83400c960 comp rx=0 tx=0).stop 2023-10-30T20:23:15.304 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 1 -- 172.21.15.69:0/3982173759 shutdown_connections 2023-10-30T20:23:15.304 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 1 --2- 172.21.15.69:0/3982173759 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff83006c610 0x7ff83006ead0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:15.304 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 1 --2- 172.21.15.69:0/3982173759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8440b2db0 0x7ff84406d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:15.304 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 1 --2- 172.21.15.69:0/3982173759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8440b0630 0x7ff8441dddd0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:15.304 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 1 --2- 172.21.15.69:0/3982173759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff844070cd0 0x7ff8441cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:15.305 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 1 -- 172.21.15.69:0/3982173759 >> 172.21.15.69:0/3982173759 conn(0x7ff8440b3e10 msgr2=0x7ff8441428d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:15.305 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 1 -- 172.21.15.69:0/3982173759 shutdown_connections 2023-10-30T20:23:15.305 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 1 -- 172.21.15.69:0/3982173759 wait complete. 2023-10-30T20:23:15.305 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:15.300+0000 7ff849456640 1 librados: shutdown 2023-10-30T20:23:15.320 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T20:23:20.452 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:07.944 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:20.512 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:08.004 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:20.554 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:08.022 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:20.618 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:08.086 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:20.694 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:08.175 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:20.753 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:08.234 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:20.849 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:08.336 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:20.910 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:08.397 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:20.951 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:08.450 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:21.011 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:08.509 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:21.054 INFO:tasks.ceph.osd.4.smithi183.stderr:2023-10-30T20:23:21.047+0000 5dc2340 -1 Falling back to public interface 2023-10-30T20:23:21.074 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:08.559 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:21.091 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:08.582 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:21.134 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:08.619 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:21.137 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:08.640 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:21.152 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:08.643 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:21.167 INFO:tasks.ceph.osd.6.smithi183.stderr:2023-10-30T20:23:21.160+0000 5dc2340 -1 Falling back to public interface 2023-10-30T20:23:21.196 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:08.698 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:21.321 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:23:21.314+0000 5dc2340 -1 Falling back to public interface 2023-10-30T20:23:21.494 INFO:tasks.ceph.osd.5.smithi183.stderr:2023-10-30T20:23:21.481+0000 5dc2340 -1 Falling back to public interface 2023-10-30T20:23:21.541 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:09.034 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:21.557 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-10-30T20:23:21.550+0000 5dc2340 -1 Falling back to public interface 2023-10-30T20:23:21.633 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:09.101 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:21.680 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-10-30T20:23:21.673+0000 5dc2340 -1 Falling back to public interface 2023-10-30T20:23:21.728 INFO:tasks.ceph.osd.3.smithi069.stderr:2023-10-30T20:23:21.721+0000 5dc2340 -1 Falling back to public interface 2023-10-30T20:23:21.769 INFO:tasks.ceph.osd.1.smithi069.stderr:2023-10-30T20:23:21.762+0000 5dc2340 -1 Falling back to public interface 2023-10-30T20:23:21.801 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:09.282 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:21.917 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:09.410 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:21.966 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:09.453 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.016 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:09.484 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.045 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:09.544 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.127 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T20:23:22.163 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:09.648 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.177 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:09.658 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.217 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.216+0000 7fd7f9fc9640 10 monclient: get_monmap_and_config 2023-10-30T20:23:22.217 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.216+0000 7fd7f9fc9640 10 monclient: build_initial_monmap 2023-10-30T20:23:22.217 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.216+0000 7fd7f9fc9640 10 monclient: monmap: 2023-10-30T20:23:22.218 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-10-30T20:23:22.218 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:22.218 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:23:22.217094+0000 2023-10-30T20:23:22.218 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:23:22.217094+0000 2023-10-30T20:23:22.219 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:22.219 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:22.219 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:22.219 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:22.219 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:22.220 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:22.220 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:22.220 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:22.221 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:09.710 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.221 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.216+0000 7fd7f9fc9640 1 Processor -- start 2023-10-30T20:23:22.221 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.216+0000 7fd7f9fc9640 1 -- start start 2023-10-30T20:23:22.221 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.216+0000 7fd7f9fc9640 10 monclient: init 2023-10-30T20:23:22.222 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:09.709 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.224 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.217+0000 7fd7f9fc9640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:22.224 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.217+0000 7fd7f9fc9640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-10-30T20:23:22.224 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.217+0000 7fd7f9fc9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd7f40a8ca0 0x7fd7f4141f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:22.225 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.217+0000 7fd7f9fc9640 10 monclient(hunting): picked mon.noname-c con 0x7fd7f40a8ca0 addr v2:172.21.15.183:3300/0 2023-10-30T20:23:22.225 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.217+0000 7fd7f9fc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7f4142450 0x7fd7f413ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:22.225 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.217+0000 7fd7f9fc9640 10 monclient(hunting): picked mon.noname-b con 0x7fd7f4142450 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:22.225 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.217+0000 7fd7f9fc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7f40a9080 addr v1:172.21.15.183:6789/0 2023-10-30T20:23:22.225 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.217+0000 7fd7f9fc9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f40a9540 con 0x7fd7f40a9080 2023-10-30T20:23:22.225 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.217+0000 7fd7f9fc9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:22.226 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.217+0000 7fd7f9fc9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7f4067a30 con 0x7fd7f40a8ca0 2023-10-30T20:23:22.226 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.217+0000 7fd7f9fc9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:22.226 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.217+0000 7fd7f9fc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f40aa770 con 0x7fd7f4142450 2023-10-30T20:23:22.226 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.217+0000 7fd7f9fc9640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:22.226 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.217+0000 7fd7f9fc9640 10 monclient(hunting): authenticate will time out at 1525.506294s 2023-10-30T20:23:22.227 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f2ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7f4142450 0x7fd7f413ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:22.227 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f37fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd7f40a8ca0 0x7fd7f4141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:22.227 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f2ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7f4142450 0x7fd7f413ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:38946/0 (socket says 172.21.15.69:38946) 2023-10-30T20:23:22.227 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f2ffd640 1 -- 172.21.15.69:0/1777819257 learned_addr learned my addr 172.21.15.69:0/1777819257 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:23:22.227 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f2ffd640 10 monclient(hunting): get_auth_request con 0x7fd7f4142450 auth_method 0 2023-10-30T20:23:22.227 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:22.228 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:22.228 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:22.228 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:22.228 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:22.228 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:22.229 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f37fe640 10 monclient(hunting): get_auth_request con 0x7fd7f40a8ca0 auth_method 0 2023-10-30T20:23:22.229 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:22.229 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:22.229 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.218+0000 7fd7f37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:22.229 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.219+0000 7fd7f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:22.230 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.219+0000 7fd7f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:22.230 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.219+0000 7fd7f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:22.230 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.220+0000 7fd7f2ffd640 10 monclient(hunting): handle_auth_done global_id 4172 payload 274 2023-10-30T20:23:22.230 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.220+0000 7fd7f2ffd640 1 -- 172.21.15.69:0/1777819257 >> v2:172.21.15.183:3300/0 conn(0x7fd7f40a8ca0 msgr2=0x7fd7f4141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:22.230 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.220+0000 7fd7f2ffd640 1 --2- 172.21.15.69:0/1777819257 >> v2:172.21.15.183:3300/0 conn(0x7fd7f40a8ca0 0x7fd7f4141f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:22.230 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.220+0000 7fd7f2ffd640 1 -- 172.21.15.69:0/1777819257 >> v1:172.21.15.183:6789/0 conn(0x7fd7f40a9080 legacy=0x7fd7f4144dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:22.231 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.220+0000 7fd7f2ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:23:22.231 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.220+0000 7fd7f2ffd640 1 monclient: found mon.noname-b 2023-10-30T20:23:22.231 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.220+0000 7fd7f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:22.231 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.220+0000 7fd7f2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:23:22.231 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.220+0000 7fd7f2ffd640 1 -- 172.21.15.69:0/1777819257 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f40b39e0 con 0x7fd7f4142450 2023-10-30T20:23:22.232 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.222+0000 7fd7f2ffd640 1 --2- 172.21.15.69:0/1777819257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7f4142450 0x7fd7f413ec30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8009870 tx=0x7fd7e802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=538200b3a9f16aa4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:22.235 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f27fc640 1 -- 172.21.15.69:0/1777819257 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8031640 con 0x7fd7f4142450 2023-10-30T20:23:22.236 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:22.236 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:23:22.236 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f27fc640 10 monclient: dump: 2023-10-30T20:23:22.236 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:22.236 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:22.237 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:22.237 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:22.237 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:22.237 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:22.237 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:22.238 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:22.238 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:22.238 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:22.238 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f27fc640 10 monclient: _finish_auth 0 2023-10-30T20:23:22.238 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:23:22.239 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:22.239 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f9fc9640 5 monclient: authenticate success, global_id 4172 2023-10-30T20:23:22.239 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f27fc640 1 -- 172.21.15.69:0/1777819257 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e80317b0 con 0x7fd7f4142450 2023-10-30T20:23:22.239 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:22.239 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f9fc9640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:22.240 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f27fc640 1 -- 172.21.15.69:0/1777819257 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8031b50 con 0x7fd7f4142450 2023-10-30T20:23:22.240 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f9fc9640 4 set_mon_vals no callback set 2023-10-30T20:23:22.240 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:22.240 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:23:22.240 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.225+0000 7fd7f27fc640 10 monclient: dump: 2023-10-30T20:23:22.241 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:22.241 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:22.241 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:22.241 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:22.242 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:22.242 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:22.242 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:22.242 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:22.242 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:22.243 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:22.243 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.226+0000 7fd7f9fc9640 10 monclient: shutdown 2023-10-30T20:23:22.243 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.226+0000 7fd7f9fc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:22.243 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.226+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/1777819257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7f4142450 msgr2=0x7fd7f413ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:22.243 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.226+0000 7fd7f9fc9640 1 --2- 172.21.15.69:0/1777819257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7f4142450 0x7fd7f413ec30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8009870 tx=0x7fd7e802b8f0 comp rx=0 tx=0).stop 2023-10-30T20:23:22.244 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.226+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/1777819257 shutdown_connections 2023-10-30T20:23:22.244 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.226+0000 7fd7f9fc9640 1 --2- 172.21.15.69:0/1777819257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7f4142450 0x7fd7f413ec30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:22.244 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.226+0000 7fd7f9fc9640 1 --2- 172.21.15.69:0/1777819257 >> v2:172.21.15.183:3300/0 conn(0x7fd7f40a8ca0 0x7fd7f4141f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:22.244 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.226+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/1777819257 >> 172.21.15.69:0/1777819257 conn(0x7fd7f40b71a0 msgr2=0x7fd7f40b75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:22.244 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.226+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/1777819257 shutdown_connections 2023-10-30T20:23:22.244 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.226+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/1777819257 wait complete. 2023-10-30T20:23:22.245 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.227+0000 7fd7f9fc9640 10 monclient: build_initial_monmap 2023-10-30T20:23:22.245 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.227+0000 7fd7f9fc9640 10 monclient: monmap: 2023-10-30T20:23:22.245 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-10-30T20:23:22.245 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:22.245 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2023-10-30T20:23:22.245 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2023-10-30T20:23:22.246 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:22.246 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-10-30T20:23:22.246 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:22.246 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:22.246 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:22.247 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:22.247 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.227+0000 7fd7f9fc9640 1 librados: starting msgr at 2023-10-30T20:23:22.247 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.227+0000 7fd7f9fc9640 1 librados: starting objecter 2023-10-30T20:23:22.247 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.228+0000 7fd7f9fc9640 1 Processor -- start 2023-10-30T20:23:22.247 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.228+0000 7fd7f9fc9640 1 -- start start 2023-10-30T20:23:22.247 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.228+0000 7fd7f9fc9640 1 librados: setting wanted keys 2023-10-30T20:23:22.248 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.228+0000 7fd7f9fc9640 1 librados: calling monclient init 2023-10-30T20:23:22.248 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.228+0000 7fd7f9fc9640 10 monclient: init 2023-10-30T20:23:22.248 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:22.248 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T20:23:22.248 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7f40a8ca0 0x7fd7f4140d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:22.249 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 10 monclient(hunting): picked mon.noname-c con 0x7fd7f40a8ca0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:22.249 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7f40a9080 0x7fd7f413df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:22.249 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 10 monclient(hunting): picked mon.noname-b con 0x7fd7f40a9080 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:22.249 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f40b4d70 0x7fd7f40b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:22.249 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 10 monclient(hunting): picked mon.noname-a con 0x7fd7f40b4d70 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:22.249 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:22.250 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f40a83d0 con 0x7fd7f40b4d70 2023-10-30T20:23:22.250 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:22.250 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f40b5fd0 con 0x7fd7f40a9080 2023-10-30T20:23:22.250 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:22.251 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f413c890 con 0x7fd7f40a8ca0 2023-10-30T20:23:22.251 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:22.251 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f9fc9640 10 monclient(hunting): authenticate will time out at 1525.518028s 2023-10-30T20:23:22.251 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f40b4d70 0x7fd7f40b63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:22.251 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.229+0000 7fd7f37fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7f40a8ca0 0x7fd7f4140d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:22.251 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f2ffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7f40a9080 0x7fd7f413df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:22.252 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f37fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7f40a8ca0 0x7fd7f4140d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:38960/0 (socket says 172.21.15.69:38960) 2023-10-30T20:23:22.252 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f37fe640 1 -- 172.21.15.69:0/2053826841 learned_addr learned my addr 172.21.15.69:0/2053826841 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:23:22.252 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f37fe640 10 monclient(hunting): get_auth_request con 0x7fd7f40a8ca0 auth_method 0 2023-10-30T20:23:22.252 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:22.252 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:22.253 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:22.253 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f3fff640 10 monclient(hunting): get_auth_request con 0x7fd7f40b4d70 auth_method 0 2023-10-30T20:23:22.253 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:22.253 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:22.253 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:22.253 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f2ffd640 10 monclient(hunting): get_auth_request con 0x7fd7f40a9080 auth_method 0 2023-10-30T20:23:22.254 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:22.254 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:22.254 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:22.254 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:22.254 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:22.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:22.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:22.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:22.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:22.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:22.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:22.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.230+0000 7fd7f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:22.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.233+0000 7fd7f2ffd640 10 monclient(hunting): handle_auth_done global_id 4189 payload 824 2023-10-30T20:23:22.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.233+0000 7fd7f2ffd640 1 -- 172.21.15.69:0/2053826841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7f40a8ca0 msgr2=0x7fd7f4140d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:22.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.233+0000 7fd7f2ffd640 1 --2- 172.21.15.69:0/2053826841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7f40a8ca0 0x7fd7f4140d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:22.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.233+0000 7fd7f2ffd640 1 -- 172.21.15.69:0/2053826841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f40b4d70 msgr2=0x7fd7f40b63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:22.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.233+0000 7fd7f2ffd640 1 --2- 172.21.15.69:0/2053826841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f40b4d70 0x7fd7f40b63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:22.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.233+0000 7fd7f2ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:23:22.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.233+0000 7fd7f2ffd640 1 monclient: found mon.noname-b 2023-10-30T20:23:22.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.233+0000 7fd7f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:22.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.233+0000 7fd7f2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:23:22.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.233+0000 7fd7f2ffd640 1 -- 172.21.15.69:0/2053826841 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f413e620 con 0x7fd7f40a9080 2023-10-30T20:23:22.258 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.235+0000 7fd7f2ffd640 1 --2- 172.21.15.69:0/2053826841 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7f40a9080 0x7fd7f413df90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd7f40a9c80 tx=0x7fd7e80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:22.259 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.235+0000 7fd7dbfff640 10 client.?.objecter ms_handle_connect 0x7fd7f40a9080 2023-10-30T20:23:22.259 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.235+0000 7fd7dbfff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:22.259 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7dbfff640 1 -- 172.21.15.69:0/2053826841 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e803ac60 con 0x7fd7f40a9080 2023-10-30T20:23:22.259 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:22.260 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7dbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:23:22.260 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7dbfff640 10 monclient: dump: 2023-10-30T20:23:22.260 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:22.260 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:22.260 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:22.261 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:22.261 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:22.261 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:22.261 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:22.261 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:22.262 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:22.262 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:22.262 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7dbfff640 10 monclient: _finish_auth 0 2023-10-30T20:23:22.262 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7dbfff640 10 monclient: _check_auth_tickets 2023-10-30T20:23:22.262 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:22.263 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7dbfff640 1 -- 172.21.15.69:0/2053826841 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e803e030 con 0x7fd7f40a9080 2023-10-30T20:23:22.263 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7dbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:22.263 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7dbfff640 1 -- 172.21.15.69:0/2053826841 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e80423f0 con 0x7fd7f40a9080 2023-10-30T20:23:22.263 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7f9fc9640 5 monclient: authenticate success, global_id 4189 2023-10-30T20:23:22.263 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7f9fc9640 10 monclient: _renew_subs 2023-10-30T20:23:22.264 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7f9fc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:22.264 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/2053826841 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f413d4b0 con 0x7fd7f40a9080 2023-10-30T20:23:22.264 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:22.264 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:23:22.264 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7dbfff640 10 monclient: dump: 2023-10-30T20:23:22.264 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:22.265 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:22.265 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:22.265 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:22.265 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:22.265 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:22.265 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:22.266 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:22.266 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:22.266 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:22.266 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7f1ffb640 4 set_mon_vals no callback set 2023-10-30T20:23:22.266 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7f9fc9640 10 client.4189.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:22.266 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7f9fc9640 10 monclient: _renew_subs 2023-10-30T20:23:22.267 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7f9fc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:22.267 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/2053826841 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7f40b6b10 con 0x7fd7f40a9080 2023-10-30T20:23:22.267 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.240+0000 7fd7f9fc9640 1 librados: init done 2023-10-30T20:23:22.267 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.241+0000 7fd7f9fc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:22.267 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.241+0000 7fd7f9fc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:22.268 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.241+0000 7fd7f9fc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:22.268 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.241+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/2053826841 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7f404e9b0 con 0x7fd7f40a9080 2023-10-30T20:23:22.268 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:09.749 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.269 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.258+0000 7fd7dbfff640 1 -- 172.21.15.69:0/2053826841 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7fd7e802f050 con 0x7fd7f40a9080 2023-10-30T20:23:22.269 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.259+0000 7fd7dbfff640 1 --2- 172.21.15.69:0/2053826841 >> v2:172.21.15.69:6800/1876307353 conn(0x7fd7e006c780 0x7fd7e006ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:22.269 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.259+0000 7fd7f37fe640 1 --2- 172.21.15.69:0/2053826841 >> v2:172.21.15.69:6800/1876307353 conn(0x7fd7e006c780 0x7fd7e006ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:22.270 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.259+0000 7fd7f37fe640 10 monclient: get_auth_request con 0x7fd7e006c780 auth_method 0 2023-10-30T20:23:22.270 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.259+0000 7fd7f37fe640 1 --2- 172.21.15.69:0/2053826841 >> v2:172.21.15.69:6800/1876307353 conn(0x7fd7e006c780 0x7fd7e006ec40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009c00 tx=0x7fd7e4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:22.270 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.259+0000 7fd7dbfff640 10 client.4189.objecter ms_handle_connect 0x7fd7e006c780 2023-10-30T20:23:22.270 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.261+0000 7fd7dbfff640 1 -- 172.21.15.69:0/2053826841 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fd7e8031e30 con 0x7fd7f40a9080 2023-10-30T20:23:22.270 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.261+0000 7fd7dbfff640 10 client.4189.objecter ms_dispatch 0x7fd7f4000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T20:23:22.271 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.261+0000 7fd7dbfff640 3 client.4189.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T20:23:22.271 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.261+0000 7fd7dbfff640 3 client.4189.objecter handle_osd_map decoding full epoch 10 2023-10-30T20:23:22.271 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.261+0000 7fd7dbfff640 20 client.4189.objecter dump_active .. 0 homeless 2023-10-30T20:23:22.312 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:09.779 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.352 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:09.839 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.420 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:09.919 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.468 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:09.949 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.492 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:09.985 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.496 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.495+0000 7fd7dbfff640 1 -- 172.21.15.69:0/2053826841 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd7e8079750 con 0x7fd7f40a9080 2023-10-30T20:23:22.497 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.495+0000 7fd7dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:22.497 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.495+0000 7fd7dbfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:22.526 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:10.019 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.552 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.037 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.583 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:10.051 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.605 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:10.096 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.643 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:10.130 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.650 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.152 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.674 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.673+0000 7fd7f9fc9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:22.674 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.673+0000 7fd7f9fc9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:22.675 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.673+0000 7fd7f9fc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:23:22.675 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.673+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/2053826841 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7f40b6b10 con 0x7fd7f40a9080 2023-10-30T20:23:22.712 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.211 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.716 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.715+0000 7fd7dbfff640 1 -- 172.21.15.69:0/2053826841 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fd7e807d7f0 con 0x7fd7f40a9080 2023-10-30T20:23:22.716 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.715+0000 7fd7dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:22.717 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.715+0000 7fd7dbfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:22.717 INFO:teuthology.misc.health.smithi069.stdout: 2023-10-30T20:23:22.717 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"32c1a723-7694-47d8-8b37-b1257dfe1796","created":"2023-10-30T20:23:06.559937+0000","modified":"2023-10-30T20:23:12.023084+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T20:23:12.023084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af10bc15-318a-4349-af76-32a30c8acf80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d0d32a9a-e24a-4d0c-9b90-125485d70672","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"508f2c3a-9e54-46aa-b6c0-52fad97fd341","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"003a2613-1050-4a96-b927-0e5983687319","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"678fc58e-ed6a-4fd0-9825-93297e3af6f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f945159a-a53c-48ba-97df-9374f9e56ffd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8184f59d-15e2-4bdc-90e3-fe1e231ca125","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T20:23:22.719 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 10 librados: watch_flush enter 2023-10-30T20:23:22.719 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 10 librados: watch_flush exit 2023-10-30T20:23:22.719 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 20 client.4189.objecter shutdown clearing up homeless session... 2023-10-30T20:23:22.719 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 10 client.4189.objecter successfully canceled tick 2023-10-30T20:23:22.720 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/2053826841 >> v2:172.21.15.69:6800/1876307353 conn(0x7fd7e006c780 msgr2=0x7fd7e006ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:22.720 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 1 --2- 172.21.15.69:0/2053826841 >> v2:172.21.15.69:6800/1876307353 conn(0x7fd7e006c780 0x7fd7e006ec40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009c00 tx=0x7fd7e4009190 comp rx=0 tx=0).stop 2023-10-30T20:23:22.720 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 10 monclient: shutdown 2023-10-30T20:23:22.720 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:22.720 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/2053826841 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7f40a9080 msgr2=0x7fd7f413df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:22.721 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 1 --2- 172.21.15.69:0/2053826841 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7f40a9080 0x7fd7f413df90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd7f40a9c80 tx=0x7fd7e80046c0 comp rx=0 tx=0).stop 2023-10-30T20:23:22.721 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/2053826841 shutdown_connections 2023-10-30T20:23:22.721 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 1 --2- 172.21.15.69:0/2053826841 >> v2:172.21.15.69:6800/1876307353 conn(0x7fd7e006c780 0x7fd7e006ec40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:22.721 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 1 --2- 172.21.15.69:0/2053826841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f40b4d70 0x7fd7f40b63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:22.721 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 1 --2- 172.21.15.69:0/2053826841 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7f40a9080 0x7fd7f413df90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:22.722 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 1 --2- 172.21.15.69:0/2053826841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7f40a8ca0 0x7fd7f4140d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:22.722 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/2053826841 >> 172.21.15.69:0/2053826841 conn(0x7fd7f40b71a0 msgr2=0x7fd7f40b7580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:22.722 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/2053826841 shutdown_connections 2023-10-30T20:23:22.722 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 1 -- 172.21.15.69:0/2053826841 wait complete. 2023-10-30T20:23:22.722 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:22.717+0000 7fd7f9fc9640 1 librados: shutdown 2023-10-30T20:23:22.729 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T20:23:22.736 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:10.217 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.793 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:10.286 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.854 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:10.322 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.872 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.357 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.901 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:10.392 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.906 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:10.394 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.943 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.446 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:22.977 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.476 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.000 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:10.481 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.061 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:10.554 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.119 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:10.587 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.143 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.628 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.168 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:10.658 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.171 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:10.658 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.208 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.710 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.241 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.740 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.265 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:10.746 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.333 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:10.825 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.394 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:10.862 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.414 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.899 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.435 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:10.922 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.437 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:10.928 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.471 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.974 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.506 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:11.005 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.529 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:11.010 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.601 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:11.094 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.658 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:11.126 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.678 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:11.163 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.700 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:11.187 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.703 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:11.194 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.736 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:11.238 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.774 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:11.273 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.797 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:11.278 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.922 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:11.390 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.943 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:11.428 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.963 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:11.450 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:23.971 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:11.462 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.020 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:11.522 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.053 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:11.552 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.077 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:11.557 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.089 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:11.582 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.119 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:11.611 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.221 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:11.707 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.253 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:11.744 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.257 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:11.744 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.309 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:11.812 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.344 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:11.843 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.455 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:11.922 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.500 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:11.968 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.513 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:11.998 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.542 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:12.033 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.600 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:12.102 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.678 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:12.159 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.732 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:12.212 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.803 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:12.290 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.834 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:12.321 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.902 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:12.401 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:24.933 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:12.432 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:25.080 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:12.566 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:25.113 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:12.599 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:25.122 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:12.613 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:25.157 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:12.648 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:25.188 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:12.690 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:25.219 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:12.721 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:29.535 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T20:23:29.631 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.629+0000 7fe8d6933640 10 monclient: get_monmap_and_config 2023-10-30T20:23:29.631 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.629+0000 7fe8d6933640 10 monclient: build_initial_monmap 2023-10-30T20:23:29.632 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.629+0000 7fe8d6933640 10 monclient: monmap: 2023-10-30T20:23:29.632 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-10-30T20:23:29.632 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:29.632 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:23:29.630744+0000 2023-10-30T20:23:29.633 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:23:29.630744+0000 2023-10-30T20:23:29.633 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:29.633 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:29.633 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:29.634 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:29.634 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:29.634 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:29.634 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:29.634 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:29.638 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.630+0000 7fe8d6933640 1 Processor -- start 2023-10-30T20:23:29.639 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.630+0000 7fe8d6933640 1 -- start start 2023-10-30T20:23:29.639 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.630+0000 7fe8d6933640 10 monclient: init 2023-10-30T20:23:29.639 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.631+0000 7fe8d6933640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:29.639 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.631+0000 7fe8d6933640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-10-30T20:23:29.639 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.631+0000 7fe8d6933640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe8d00a8ca0 0x7fe8d0141f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:29.640 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.631+0000 7fe8d6933640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d00a8ca0 addr v2:172.21.15.69:3300/0 2023-10-30T20:23:29.640 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.631+0000 7fe8d6933640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8d0142450 addr v1:172.21.15.183:6789/0 2023-10-30T20:23:29.640 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.631+0000 7fe8d6933640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8d00a9080 0x7fe8d014c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:29.641 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.631+0000 7fe8d6933640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d00a9080 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:29.641 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.631+0000 7fe8d6933640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00a9540 con 0x7fe8d0142450 2023-10-30T20:23:29.641 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.631+0000 7fe8d6933640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:29.641 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.631+0000 7fe8d6933640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8d0067a30 con 0x7fe8d00a8ca0 2023-10-30T20:23:29.641 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.631+0000 7fe8d6933640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:29.642 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.631+0000 7fe8d6933640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d00aa770 con 0x7fe8d00a9080 2023-10-30T20:23:29.642 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.631+0000 7fe8d6933640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:29.642 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.631+0000 7fe8d6933640 10 monclient(hunting): authenticate will time out at 1532.920149s 2023-10-30T20:23:29.643 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8d4ed0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8d00a9080 0x7fe8d014c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:29.643 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8d4ed0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8d00a9080 0x7fe8d014c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:35674/0 (socket says 172.21.15.69:35674) 2023-10-30T20:23:29.643 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8d4ed0640 1 -- 172.21.15.69:0/3616008182 learned_addr learned my addr 172.21.15.69:0/3616008182 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:23:29.643 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8d4ed0640 10 monclient(hunting): get_auth_request con 0x7fe8d00a9080 auth_method 0 2023-10-30T20:23:29.644 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8d4ed0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:29.644 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8d4ed0640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:29.644 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8d4ed0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:29.644 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8d4ed0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:29.645 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8d4ed0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:29.645 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8d4ed0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:29.645 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8cffff640 1 --2- 172.21.15.69:0/3616008182 >> v2:172.21.15.69:3300/0 conn(0x7fe8d00a8ca0 0x7fe8d0141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:29.645 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8cffff640 10 monclient(hunting): get_auth_request con 0x7fe8d00a8ca0 auth_method 0 2023-10-30T20:23:29.646 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:29.646 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8cffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:29.646 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.632+0000 7fe8cffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:29.646 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.633+0000 7fe8cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:29.647 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.633+0000 7fe8cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:29.647 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.633+0000 7fe8cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:29.647 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.635+0000 7fe8d4ed0640 10 monclient(hunting): handle_auth_done global_id 4178 payload 274 2023-10-30T20:23:29.647 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.635+0000 7fe8d4ed0640 1 -- 172.21.15.69:0/3616008182 >> v2:172.21.15.69:3300/0 conn(0x7fe8d00a8ca0 msgr2=0x7fe8d0141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:29.648 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.635+0000 7fe8d4ed0640 1 --2- 172.21.15.69:0/3616008182 >> v2:172.21.15.69:3300/0 conn(0x7fe8d00a8ca0 0x7fe8d0141f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:29.648 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.635+0000 7fe8d4ed0640 1 -- 172.21.15.69:0/3616008182 >> v1:172.21.15.183:6789/0 conn(0x7fe8d0142450 legacy=0x7fe8d013ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:29.648 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.635+0000 7fe8d4ed0640 10 monclient: _finish_hunting 0 2023-10-30T20:23:29.649 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.635+0000 7fe8d4ed0640 1 monclient: found mon.noname-b 2023-10-30T20:23:29.649 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.635+0000 7fe8d4ed0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:29.649 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.635+0000 7fe8d4ed0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:23:29.650 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.635+0000 7fe8d4ed0640 1 -- 172.21.15.69:0/3616008182 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d00b39e0 con 0x7fe8d00a9080 2023-10-30T20:23:29.650 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.637+0000 7fe8d4ed0640 1 --2- 172.21.15.69:0/3616008182 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8d00a9080 0x7fe8d014c160 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009e50 tx=0x7fe8c802bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa62898dcce34004 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:29.658 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8ceffd640 1 -- 172.21.15.69:0/3616008182 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c802caa0 con 0x7fe8d00a9080 2023-10-30T20:23:29.658 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:29.658 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8ceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:23:29.658 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8ceffd640 10 monclient: dump: 2023-10-30T20:23:29.659 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:29.659 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:29.659 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:29.659 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:29.659 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:29.660 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:29.660 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:29.660 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:29.660 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:29.660 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:29.661 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8ceffd640 10 monclient: _finish_auth 0 2023-10-30T20:23:29.661 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8ceffd640 10 monclient: _check_auth_tickets 2023-10-30T20:23:29.661 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:29.661 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8ceffd640 1 -- 172.21.15.69:0/3616008182 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c802cc10 con 0x7fe8d00a9080 2023-10-30T20:23:29.661 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8ceffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:29.662 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8ceffd640 1 -- 172.21.15.69:0/3616008182 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c80343f0 con 0x7fe8d00a9080 2023-10-30T20:23:29.662 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:29.662 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:23:29.662 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8ceffd640 10 monclient: dump: 2023-10-30T20:23:29.663 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:29.663 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:29.663 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:29.663 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:29.663 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:29.664 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:29.664 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:29.664 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:29.664 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:29.664 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:29.665 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8d6933640 5 monclient: authenticate success, global_id 4178 2023-10-30T20:23:29.665 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8d6933640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:29.665 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.640+0000 7fe8d6933640 4 set_mon_vals no callback set 2023-10-30T20:23:29.665 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.641+0000 7fe8d6933640 10 monclient: shutdown 2023-10-30T20:23:29.666 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.641+0000 7fe8d6933640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:29.666 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.641+0000 7fe8d6933640 1 -- 172.21.15.69:0/3616008182 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8d00a9080 msgr2=0x7fe8d014c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:29.666 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.641+0000 7fe8d6933640 1 --2- 172.21.15.69:0/3616008182 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8d00a9080 0x7fe8d014c160 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009e50 tx=0x7fe8c802bf00 comp rx=0 tx=0).stop 2023-10-30T20:23:29.666 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.641+0000 7fe8d6933640 1 -- 172.21.15.69:0/3616008182 shutdown_connections 2023-10-30T20:23:29.666 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.641+0000 7fe8d6933640 1 --2- 172.21.15.69:0/3616008182 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8d00a9080 0x7fe8d014c160 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:29.667 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.641+0000 7fe8d6933640 1 --2- 172.21.15.69:0/3616008182 >> v2:172.21.15.69:3300/0 conn(0x7fe8d00a8ca0 0x7fe8d0141f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:29.667 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.641+0000 7fe8d6933640 1 -- 172.21.15.69:0/3616008182 >> 172.21.15.69:0/3616008182 conn(0x7fe8d00b71a0 msgr2=0x7fe8d00b75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:29.667 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.641+0000 7fe8d6933640 1 -- 172.21.15.69:0/3616008182 shutdown_connections 2023-10-30T20:23:29.667 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.641+0000 7fe8d6933640 1 -- 172.21.15.69:0/3616008182 wait complete. 2023-10-30T20:23:29.667 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.642+0000 7fe8d6933640 10 monclient: build_initial_monmap 2023-10-30T20:23:29.668 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.642+0000 7fe8d6933640 10 monclient: monmap: 2023-10-30T20:23:29.668 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-10-30T20:23:29.668 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:29.668 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2023-10-30T20:23:29.668 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2023-10-30T20:23:29.669 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:29.669 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-10-30T20:23:29.669 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:29.669 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:29.670 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:29.670 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:29.670 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.642+0000 7fe8d6933640 1 librados: starting msgr at 2023-10-30T20:23:29.670 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.642+0000 7fe8d6933640 1 librados: starting objecter 2023-10-30T20:23:29.670 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 1 Processor -- start 2023-10-30T20:23:29.671 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 1 -- start start 2023-10-30T20:23:29.671 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 1 librados: setting wanted keys 2023-10-30T20:23:29.671 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 1 librados: calling monclient init 2023-10-30T20:23:29.671 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 10 monclient: init 2023-10-30T20:23:29.671 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:29.672 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:23:29.672 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8d00a8ca0 0x7fe8d013a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:29.672 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d00a8ca0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:29.673 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8d00a9080 0x7fe8d0133c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:29.673 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d00a9080 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:29.673 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8d01341a0 0x7fe8d013b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:29.673 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d01341a0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:29.674 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:29.674 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d00b37f0 con 0x7fe8d00a8ca0 2023-10-30T20:23:29.674 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:29.674 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d0133740 con 0x7fe8d00a9080 2023-10-30T20:23:29.675 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:29.675 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d00b6e60 con 0x7fe8d01341a0 2023-10-30T20:23:29.675 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:29.675 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.643+0000 7fe8d6933640 10 monclient(hunting): authenticate will time out at 1532.932697s 2023-10-30T20:23:29.676 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.644+0000 7fe8d4ed0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8d01341a0 0x7fe8d013b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:29.676 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.644+0000 7fe8d4ed0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8d01341a0 0x7fe8d013b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:35688/0 (socket says 172.21.15.69:35688) 2023-10-30T20:23:29.676 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.644+0000 7fe8d4ed0640 1 -- 172.21.15.69:0/1553706134 learned_addr learned my addr 172.21.15.69:0/1553706134 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:23:29.677 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.644+0000 7fe8d4ed0640 10 monclient(hunting): get_auth_request con 0x7fe8d01341a0 auth_method 0 2023-10-30T20:23:29.677 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.644+0000 7fe8d4ed0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:29.677 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.644+0000 7fe8d4ed0640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:29.677 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.644+0000 7fe8d4ed0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:29.678 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.644+0000 7fe8cffff640 1 --2- 172.21.15.69:0/1553706134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8d00a8ca0 0x7fe8d013a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:29.678 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.644+0000 7fe8cf7fe640 1 --2- 172.21.15.69:0/1553706134 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8d00a9080 0x7fe8d0133c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:29.678 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8d4ed0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:29.678 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8d4ed0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:29.679 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8d4ed0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:29.679 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8cffff640 10 monclient(hunting): get_auth_request con 0x7fe8d00a8ca0 auth_method 0 2023-10-30T20:23:29.679 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:29.679 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8cffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:29.680 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8cffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:29.680 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8cf7fe640 10 monclient(hunting): get_auth_request con 0x7fe8d00a9080 auth_method 0 2023-10-30T20:23:29.680 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:29.680 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8cf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:29.680 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8cf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:29.681 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:29.681 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:29.681 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:29.681 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:29.681 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:29.682 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.645+0000 7fe8cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:29.682 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.647+0000 7fe8d4ed0640 10 monclient(hunting): handle_auth_done global_id 4181 payload 824 2023-10-30T20:23:29.682 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.647+0000 7fe8d4ed0640 1 -- 172.21.15.69:0/1553706134 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8d00a9080 msgr2=0x7fe8d0133c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:29.683 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.647+0000 7fe8d4ed0640 1 --2- 172.21.15.69:0/1553706134 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8d00a9080 0x7fe8d0133c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:29.683 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.647+0000 7fe8d4ed0640 1 -- 172.21.15.69:0/1553706134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8d00a8ca0 msgr2=0x7fe8d013a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:29.683 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.647+0000 7fe8d4ed0640 1 --2- 172.21.15.69:0/1553706134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8d00a8ca0 0x7fe8d013a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:29.684 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.647+0000 7fe8d4ed0640 10 monclient: _finish_hunting 0 2023-10-30T20:23:29.684 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.647+0000 7fe8d4ed0640 1 monclient: found mon.noname-c 2023-10-30T20:23:29.684 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.647+0000 7fe8d4ed0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:29.685 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.647+0000 7fe8d4ed0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:23:29.685 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.647+0000 7fe8d4ed0640 1 -- 172.21.15.69:0/1553706134 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d013ace0 con 0x7fe8d01341a0 2023-10-30T20:23:29.685 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.650+0000 7fe8d4ed0640 1 --2- 172.21.15.69:0/1553706134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8d01341a0 0x7fe8d013b490 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009e50 tx=0x7fe8c8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:29.686 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.650+0000 7fe8ccff9640 10 client.?.objecter ms_handle_connect 0x7fe8d01341a0 2023-10-30T20:23:29.686 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.650+0000 7fe8ccff9640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:29.686 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8ccff9640 1 -- 172.21.15.69:0/1553706134 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c803e450 con 0x7fe8d01341a0 2023-10-30T20:23:29.686 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:29.687 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8ccff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:23:29.687 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8ccff9640 10 monclient: dump: 2023-10-30T20:23:29.687 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:29.687 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:29.688 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:29.688 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:29.688 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:29.688 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:29.688 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:29.689 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:29.689 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:29.689 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:29.689 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8ccff9640 10 monclient: _finish_auth 0 2023-10-30T20:23:29.690 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8ccff9640 10 monclient: _check_auth_tickets 2023-10-30T20:23:29.690 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:29.690 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8ccff9640 1 -- 172.21.15.69:0/1553706134 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c803e600 con 0x7fe8d01341a0 2023-10-30T20:23:29.690 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8d6933640 5 monclient: authenticate success, global_id 4181 2023-10-30T20:23:29.691 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8d6933640 10 monclient: _renew_subs 2023-10-30T20:23:29.691 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8d6933640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:29.691 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8d6933640 1 -- 172.21.15.69:0/1553706134 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d013b9d0 con 0x7fe8d01341a0 2023-10-30T20:23:29.691 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8ccff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:29.692 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8ccff9640 1 -- 172.21.15.69:0/1553706134 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c803e9c0 con 0x7fe8d01341a0 2023-10-30T20:23:29.692 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:29.692 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8ccff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:23:29.692 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8ccff9640 10 monclient: dump: 2023-10-30T20:23:29.693 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:29.693 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:29.693 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:29.693 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:29.693 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:29.694 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:29.694 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:29.694 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:29.694 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:29.694 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:29.695 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8ce7fc640 4 set_mon_vals no callback set 2023-10-30T20:23:29.695 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8d6933640 10 client.4181.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:29.695 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8d6933640 10 monclient: _renew_subs 2023-10-30T20:23:29.695 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8d6933640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:29.695 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8d6933640 1 -- 172.21.15.69:0/1553706134 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d01d6900 con 0x7fe8d01341a0 2023-10-30T20:23:29.696 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.653+0000 7fe8d6933640 1 librados: init done 2023-10-30T20:23:29.696 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.654+0000 7fe8d6933640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:29.696 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.654+0000 7fe8d6933640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:29.696 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.654+0000 7fe8d6933640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:29.697 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.654+0000 7fe8d6933640 1 -- 172.21.15.69:0/1553706134 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d004e9b0 con 0x7fe8d01341a0 2023-10-30T20:23:29.698 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.676+0000 7fe8ccff9640 1 -- 172.21.15.69:0/1553706134 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7fe8c802cab0 con 0x7fe8d01341a0 2023-10-30T20:23:29.698 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.677+0000 7fe8ccff9640 1 --2- 172.21.15.69:0/1553706134 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe8bc06c730 0x7fe8bc06ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:29.699 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.677+0000 7fe8cffff640 1 --2- 172.21.15.69:0/1553706134 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe8bc06c730 0x7fe8bc06ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:29.699 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.677+0000 7fe8cffff640 10 monclient: get_auth_request con 0x7fe8bc06c730 auth_method 0 2023-10-30T20:23:29.699 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.678+0000 7fe8ccff9640 10 client.4181.objecter ms_handle_connect 0x7fe8bc06c730 2023-10-30T20:23:29.699 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.678+0000 7fe8cffff640 1 --2- 172.21.15.69:0/1553706134 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe8bc06c730 0x7fe8bc06ebf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0009b40 tx=0x7fe8c0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:29.699 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.680+0000 7fe8ccff9640 1 -- 172.21.15.69:0/1553706134 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fe8c807aba0 con 0x7fe8d01341a0 2023-10-30T20:23:29.700 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.680+0000 7fe8ccff9640 10 client.4181.objecter ms_dispatch 0x7fe8d0000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T20:23:29.700 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.680+0000 7fe8ccff9640 3 client.4181.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T20:23:29.700 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.680+0000 7fe8ccff9640 3 client.4181.objecter handle_osd_map decoding full epoch 10 2023-10-30T20:23:29.700 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.680+0000 7fe8ccff9640 20 client.4181.objecter dump_active .. 0 homeless 2023-10-30T20:23:29.840 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:17.332 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:29.892 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:17.360 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:29.989 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.987+0000 7fe8ccff9640 1 -- 172.21.15.69:0/1553706134 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe8c807ec40 con 0x7fe8d01341a0 2023-10-30T20:23:29.990 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.987+0000 7fe8ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:29.990 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:29.987+0000 7fe8ccff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:30.188 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.187+0000 7fe8d6933640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:30.189 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.187+0000 7fe8d6933640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:30.189 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.187+0000 7fe8d6933640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:30.189 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.187+0000 7fe8d6933640 1 -- 172.21.15.69:0/1553706134 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8d00b39e0 con 0x7fe8d01341a0 2023-10-30T20:23:30.203 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.202+0000 7fe8ccff9640 1 -- 172.21.15.69:0/1553706134 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe8c8082df0 con 0x7fe8d01341a0 2023-10-30T20:23:30.203 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.202+0000 7fe8ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:30.204 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.202+0000 7fe8ccff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:30.204 INFO:teuthology.misc.health.smithi069.stdout: 2023-10-30T20:23:30.204 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"32c1a723-7694-47d8-8b37-b1257dfe1796","created":"2023-10-30T20:23:06.559937+0000","modified":"2023-10-30T20:23:12.023084+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T20:23:12.023084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af10bc15-318a-4349-af76-32a30c8acf80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d0d32a9a-e24a-4d0c-9b90-125485d70672","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"508f2c3a-9e54-46aa-b6c0-52fad97fd341","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"003a2613-1050-4a96-b927-0e5983687319","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"678fc58e-ed6a-4fd0-9825-93297e3af6f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f945159a-a53c-48ba-97df-9374f9e56ffd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8184f59d-15e2-4bdc-90e3-fe1e231ca125","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T20:23:30.206 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 10 librados: watch_flush enter 2023-10-30T20:23:30.206 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 10 librados: watch_flush exit 2023-10-30T20:23:30.207 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 20 client.4181.objecter shutdown clearing up homeless session... 2023-10-30T20:23:30.207 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 10 client.4181.objecter successfully canceled tick 2023-10-30T20:23:30.207 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 1 -- 172.21.15.69:0/1553706134 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe8bc06c730 msgr2=0x7fe8bc06ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:30.207 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 1 --2- 172.21.15.69:0/1553706134 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe8bc06c730 0x7fe8bc06ebf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0009b40 tx=0x7fe8c0009190 comp rx=0 tx=0).stop 2023-10-30T20:23:30.207 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 10 monclient: shutdown 2023-10-30T20:23:30.208 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:30.208 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 1 -- 172.21.15.69:0/1553706134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8d01341a0 msgr2=0x7fe8d013b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:30.208 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 1 --2- 172.21.15.69:0/1553706134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8d01341a0 0x7fe8d013b490 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009e50 tx=0x7fe8c8004a40 comp rx=0 tx=0).stop 2023-10-30T20:23:30.208 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 1 -- 172.21.15.69:0/1553706134 shutdown_connections 2023-10-30T20:23:30.208 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 1 --2- 172.21.15.69:0/1553706134 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe8bc06c730 0x7fe8bc06ebf0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:30.209 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 1 --2- 172.21.15.69:0/1553706134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8d01341a0 0x7fe8d013b490 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:30.209 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 1 --2- 172.21.15.69:0/1553706134 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8d00a9080 0x7fe8d0133c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:30.209 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 1 --2- 172.21.15.69:0/1553706134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8d00a8ca0 0x7fe8d013a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:30.209 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 1 -- 172.21.15.69:0/1553706134 >> 172.21.15.69:0/1553706134 conn(0x7fe8d00b71a0 msgr2=0x7fe8d0137b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:30.209 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 1 -- 172.21.15.69:0/1553706134 shutdown_connections 2023-10-30T20:23:30.209 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.204+0000 7fe8d6933640 1 -- 172.21.15.69:0/1553706134 wait complete. 2023-10-30T20:23:30.210 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:30.205+0000 7fe8d6933640 1 librados: shutdown 2023-10-30T20:23:30.217 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T20:23:30.382 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:17.867 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:30.472 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:17.971 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:30.513 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:17.994 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:30.647 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:18.138 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:30.684 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:18.186 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:30.949 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:18.436 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:35.311 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:22.803 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:35.317 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:22.809 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:35.826 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:23.311 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:35.832 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:23.317 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:36.061 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:23.560 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:36.067 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:23.566 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:36.184 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:23.652 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:36.188 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:23.656 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:36.305 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:23.785 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:36.311 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:23.792 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:36.545 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:24.036 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:36.549 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:24.040 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:36.650 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:24.152 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:36.656 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:24.158 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:36.800 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:24.287 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:36.805 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:24.293 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:37.023 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T20:23:37.156 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.154+0000 7f623440e640 10 monclient: get_monmap_and_config 2023-10-30T20:23:37.156 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.154+0000 7f623440e640 10 monclient: build_initial_monmap 2023-10-30T20:23:37.156 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.155+0000 7f623440e640 10 monclient: monmap: 2023-10-30T20:23:37.156 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-10-30T20:23:37.157 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:37.157 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:23:37.155974+0000 2023-10-30T20:23:37.157 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:23:37.155974+0000 2023-10-30T20:23:37.157 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:37.157 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:37.157 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:37.158 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:37.158 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:37.158 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:37.158 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:37.158 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:37.160 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.155+0000 7f623440e640 1 Processor -- start 2023-10-30T20:23:37.160 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.155+0000 7f623440e640 1 -- start start 2023-10-30T20:23:37.160 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.155+0000 7f623440e640 10 monclient: init 2023-10-30T20:23:37.161 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.156+0000 7f623440e640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:37.161 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.156+0000 7f623440e640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-10-30T20:23:37.161 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.156+0000 7f623440e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f622c0a8ca0 0x7f622c141f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:37.161 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.156+0000 7f623440e640 10 monclient(hunting): picked mon.noname-b con 0x7f622c0a8ca0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:37.161 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.156+0000 7f623440e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f622c142450 0x7f622c13ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:37.162 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.156+0000 7f623440e640 10 monclient(hunting): picked mon.noname-c con 0x7f622c142450 addr v2:172.21.15.183:3300/0 2023-10-30T20:23:37.162 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.157+0000 7f623440e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f622c0a9080 addr v1:172.21.15.69:6789/0 2023-10-30T20:23:37.162 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.157+0000 7f623440e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622c0a9540 con 0x7f622c0a9080 2023-10-30T20:23:37.162 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.157+0000 7f623440e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:37.162 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.157+0000 7f623440e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f622c067a30 con 0x7f622c142450 2023-10-30T20:23:37.162 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.157+0000 7f623440e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:37.163 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.157+0000 7f623440e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f622c0aa770 con 0x7f622c0a8ca0 2023-10-30T20:23:37.163 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.157+0000 7f623440e640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:37.163 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.157+0000 7f623440e640 10 monclient(hunting): authenticate will time out at 1540.445913s 2023-10-30T20:23:37.163 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.157+0000 7f62321aa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f622c0a8ca0 0x7f622c141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:37.163 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.157+0000 7f62319a9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f622c142450 0x7f622c13ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:37.164 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.157+0000 7f62329ab640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f622c0a9080 0x7f622c144dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:60182/0 (socket says 172.21.15.69:60182) 2023-10-30T20:23:37.164 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.157+0000 7f62329ab640 1 -- 172.21.15.69:0/3611188868 learned_addr learned my addr 172.21.15.69:0/3611188868 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:23:37.164 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.158+0000 7f62321aa640 10 monclient(hunting): get_auth_request con 0x7f622c0a8ca0 auth_method 0 2023-10-30T20:23:37.164 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.158+0000 7f62321aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:37.164 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.158+0000 7f62321aa640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:37.164 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.158+0000 7f62321aa640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:37.165 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.158+0000 7f62319a9640 10 monclient(hunting): get_auth_request con 0x7f622c142450 auth_method 0 2023-10-30T20:23:37.165 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.158+0000 7f62319a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:37.165 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.158+0000 7f62319a9640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:37.165 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.158+0000 7f62319a9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:37.165 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.158+0000 7f62321aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:37.165 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.158+0000 7f62321aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:37.166 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.158+0000 7f62321aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:37.166 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.160+0000 7f62319a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:37.167 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.160+0000 7f62319a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:37.167 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.160+0000 7f62319a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:37.167 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.161+0000 7f62321aa640 10 monclient(hunting): handle_auth_done global_id 4184 payload 274 2023-10-30T20:23:37.167 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.161+0000 7f62321aa640 1 -- 172.21.15.69:0/3611188868 >> v2:172.21.15.183:3300/0 conn(0x7f622c142450 msgr2=0x7f622c13ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:37.168 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.161+0000 7f62321aa640 1 --2- 172.21.15.69:0/3611188868 >> v2:172.21.15.183:3300/0 conn(0x7f622c142450 0x7f622c13ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:37.168 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.161+0000 7f62321aa640 1 -- 172.21.15.69:0/3611188868 >> v1:172.21.15.69:6789/0 conn(0x7f622c0a9080 legacy=0x7f622c144dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:37.168 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.161+0000 7f62321aa640 10 monclient: _finish_hunting 0 2023-10-30T20:23:37.168 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.161+0000 7f62321aa640 1 monclient: found mon.noname-b 2023-10-30T20:23:37.168 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.161+0000 7f62321aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:37.169 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.161+0000 7f62321aa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:23:37.169 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.161+0000 7f62321aa640 1 -- 172.21.15.69:0/3611188868 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c0b39e0 con 0x7f622c0a8ca0 2023-10-30T20:23:37.169 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.163+0000 7f62321aa640 1 --2- 172.21.15.69:0/3611188868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f622c0a8ca0 0x7f622c141f10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f62280097a0 tx=0x7f622802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ec12a883c9c9358 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:37.173 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f62311a8640 1 -- 172.21.15.69:0/3611188868 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6228031780 con 0x7f622c0a8ca0 2023-10-30T20:23:37.173 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f62311a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:37.174 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f62311a8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:23:37.174 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f62311a8640 10 monclient: dump: 2023-10-30T20:23:37.174 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:37.174 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:37.175 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:37.175 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:37.175 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:37.175 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:37.175 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:37.175 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:37.176 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:37.176 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:37.176 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f62311a8640 10 monclient: _finish_auth 0 2023-10-30T20:23:37.176 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f62311a8640 10 monclient: _check_auth_tickets 2023-10-30T20:23:37.176 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f62311a8640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:37.176 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f62311a8640 1 -- 172.21.15.69:0/3611188868 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62280318f0 con 0x7f622c0a8ca0 2023-10-30T20:23:37.177 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f62311a8640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:37.177 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f62311a8640 1 -- 172.21.15.69:0/3611188868 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6228031c90 con 0x7f622c0a8ca0 2023-10-30T20:23:37.177 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f62311a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:37.177 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f62311a8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:23:37.177 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f62311a8640 10 monclient: dump: 2023-10-30T20:23:37.177 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:37.178 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:37.178 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:37.178 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:37.178 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:37.178 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:37.178 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:37.179 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:37.179 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:37.179 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:37.179 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f623440e640 5 monclient: authenticate success, global_id 4184 2023-10-30T20:23:37.179 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f623440e640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:37.179 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.172+0000 7f623440e640 4 set_mon_vals no callback set 2023-10-30T20:23:37.184 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.173+0000 7f623440e640 10 monclient: shutdown 2023-10-30T20:23:37.184 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.173+0000 7f623440e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:37.184 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.173+0000 7f623440e640 1 -- 172.21.15.69:0/3611188868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f622c0a8ca0 msgr2=0x7f622c141f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:37.185 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.173+0000 7f623440e640 1 --2- 172.21.15.69:0/3611188868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f622c0a8ca0 0x7f622c141f10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f62280097a0 tx=0x7f622802b820 comp rx=0 tx=0).stop 2023-10-30T20:23:37.185 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.173+0000 7f623440e640 1 -- 172.21.15.69:0/3611188868 shutdown_connections 2023-10-30T20:23:37.185 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.173+0000 7f623440e640 1 --2- 172.21.15.69:0/3611188868 >> v2:172.21.15.183:3300/0 conn(0x7f622c142450 0x7f622c13ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:37.185 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.173+0000 7f623440e640 1 --2- 172.21.15.69:0/3611188868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f622c0a8ca0 0x7f622c141f10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:37.185 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.173+0000 7f623440e640 1 -- 172.21.15.69:0/3611188868 >> 172.21.15.69:0/3611188868 conn(0x7f622c0b71a0 msgr2=0x7f622c0b75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:37.186 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.173+0000 7f623440e640 1 -- 172.21.15.69:0/3611188868 shutdown_connections 2023-10-30T20:23:37.186 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.173+0000 7f623440e640 1 -- 172.21.15.69:0/3611188868 wait complete. 2023-10-30T20:23:37.186 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.174+0000 7f623440e640 10 monclient: build_initial_monmap 2023-10-30T20:23:37.186 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.174+0000 7f623440e640 10 monclient: monmap: 2023-10-30T20:23:37.186 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-10-30T20:23:37.186 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:37.187 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2023-10-30T20:23:37.187 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2023-10-30T20:23:37.187 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:37.187 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-10-30T20:23:37.187 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:37.187 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:37.188 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:37.188 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:37.188 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.175+0000 7f623440e640 1 librados: starting msgr at 2023-10-30T20:23:37.188 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.175+0000 7f623440e640 1 librados: starting objecter 2023-10-30T20:23:37.188 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.175+0000 7f623440e640 1 Processor -- start 2023-10-30T20:23:37.189 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.175+0000 7f623440e640 1 -- start start 2023-10-30T20:23:37.189 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.175+0000 7f623440e640 1 librados: setting wanted keys 2023-10-30T20:23:37.189 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.175+0000 7f623440e640 1 librados: calling monclient init 2023-10-30T20:23:37.189 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.175+0000 7f623440e640 10 monclient: init 2023-10-30T20:23:37.189 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.176+0000 7f623440e640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:37.189 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.176+0000 7f623440e640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:23:37.190 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.176+0000 7f623440e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f622c0a8ca0 0x7f622c13b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:37.190 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f623440e640 10 monclient(hunting): picked mon.noname-b con 0x7f622c0a8ca0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:37.190 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f623440e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f622c0a9080 0x7f622c0acca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:37.190 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f623440e640 10 monclient(hunting): picked mon.noname-c con 0x7f622c0a9080 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:37.190 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f623440e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f622c0ad1e0 0x7f622c1363a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:37.191 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f623440e640 10 monclient(hunting): picked mon.noname-a con 0x7f622c0ad1e0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:37.191 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f623440e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:37.191 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f623440e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c0a83d0 con 0x7f622c0ad1e0 2023-10-30T20:23:37.191 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f623440e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:37.191 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f623440e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c0ad740 con 0x7f622c0a8ca0 2023-10-30T20:23:37.191 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f623440e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:37.192 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f623440e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f622c13c150 con 0x7f622c0a9080 2023-10-30T20:23:37.192 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f62321aa640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f622c0a8ca0 0x7f622c13b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:37.192 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f623440e640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:37.192 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f623440e640 10 monclient(hunting): authenticate will time out at 1540.466519s 2023-10-30T20:23:37.192 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f62319a9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f622c0a9080 0x7f622c0acca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:37.192 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.177+0000 7f62329ab640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f622c0ad1e0 0x7f622c1363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:37.193 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62319a9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f622c0a9080 0x7f622c0acca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:35704/0 (socket says 172.21.15.69:35704) 2023-10-30T20:23:37.193 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62319a9640 1 -- 172.21.15.69:0/1671123431 learned_addr learned my addr 172.21.15.69:0/1671123431 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:23:37.193 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62319a9640 10 monclient(hunting): get_auth_request con 0x7f622c0a9080 auth_method 0 2023-10-30T20:23:37.193 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62319a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:37.193 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62319a9640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:37.194 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62319a9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:37.194 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62321aa640 10 monclient(hunting): get_auth_request con 0x7f622c0a8ca0 auth_method 0 2023-10-30T20:23:37.194 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62321aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:37.194 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62321aa640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:37.194 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62321aa640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:37.194 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62329ab640 10 monclient(hunting): get_auth_request con 0x7f622c0ad1e0 auth_method 0 2023-10-30T20:23:37.195 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62329ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:37.195 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62329ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:37.195 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62329ab640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:37.195 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62319a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:37.195 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62319a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:37.197 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62319a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:37.197 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62321aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:37.197 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62321aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:37.197 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.178+0000 7f62321aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:37.197 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.179+0000 7f62329ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:37.198 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.179+0000 7f62329ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:37.198 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.179+0000 7f62329ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:37.198 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.182+0000 7f62319a9640 10 monclient(hunting): handle_auth_done global_id 4187 payload 824 2023-10-30T20:23:37.198 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.182+0000 7f62319a9640 1 -- 172.21.15.69:0/1671123431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f622c0a8ca0 msgr2=0x7f622c13b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:37.198 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.182+0000 7f62319a9640 1 --2- 172.21.15.69:0/1671123431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f622c0a8ca0 0x7f622c13b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:37.199 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.182+0000 7f62319a9640 1 -- 172.21.15.69:0/1671123431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f622c0ad1e0 msgr2=0x7f622c1363a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:37.199 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.182+0000 7f62319a9640 1 --2- 172.21.15.69:0/1671123431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f622c0ad1e0 0x7f622c1363a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:37.199 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.182+0000 7f62319a9640 10 monclient: _finish_hunting 0 2023-10-30T20:23:37.199 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.182+0000 7f62319a9640 1 monclient: found mon.noname-c 2023-10-30T20:23:37.199 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.182+0000 7f62319a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:37.199 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.182+0000 7f62319a9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:23:37.200 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.182+0000 7f62319a9640 1 -- 172.21.15.69:0/1671123431 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c1368e0 con 0x7f622c0a9080 2023-10-30T20:23:37.201 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.185+0000 7f62319a9640 1 --2- 172.21.15.69:0/1671123431 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f622c0a9080 0x7f622c0acca0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f621c00e8a0 tx=0x7f621c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:37.201 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.185+0000 7f6222ffd640 10 client.?.objecter ms_handle_connect 0x7f622c0a9080 2023-10-30T20:23:37.202 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.185+0000 7f6222ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:37.202 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f6222ffd640 1 -- 172.21.15.69:0/1671123431 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c00c940 con 0x7f622c0a9080 2023-10-30T20:23:37.202 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f6222ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:37.202 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f6222ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:23:37.202 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f6222ffd640 10 monclient: dump: 2023-10-30T20:23:37.202 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:37.203 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:37.203 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:37.203 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:37.203 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:37.203 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:37.204 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:37.204 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:37.204 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:37.204 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:37.204 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f6222ffd640 10 monclient: _finish_auth 0 2023-10-30T20:23:37.205 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f6222ffd640 10 monclient: _check_auth_tickets 2023-10-30T20:23:37.205 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f6222ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:37.205 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f6222ffd640 1 -- 172.21.15.69:0/1671123431 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621c00d040 con 0x7f622c0a9080 2023-10-30T20:23:37.205 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f623440e640 5 monclient: authenticate success, global_id 4187 2023-10-30T20:23:37.205 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f623440e640 10 monclient: _renew_subs 2023-10-30T20:23:37.205 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f623440e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:37.206 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f623440e640 1 -- 172.21.15.69:0/1671123431 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622c136ae0 con 0x7f622c0a9080 2023-10-30T20:23:37.206 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f6222ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:37.206 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f62309a7640 4 set_mon_vals no callback set 2023-10-30T20:23:37.207 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f623440e640 10 client.4187.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:37.207 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f623440e640 10 monclient: _renew_subs 2023-10-30T20:23:37.207 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f623440e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:37.207 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f623440e640 1 -- 172.21.15.69:0/1671123431 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622c1d9db0 con 0x7f622c0a9080 2023-10-30T20:23:37.207 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f623440e640 1 librados: init done 2023-10-30T20:23:37.208 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.191+0000 7f6222ffd640 1 -- 172.21.15.69:0/1671123431 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c00ccf0 con 0x7f622c0a9080 2023-10-30T20:23:37.208 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.192+0000 7f6222ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:37.208 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.192+0000 7f6222ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:23:37.208 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.192+0000 7f6222ffd640 10 monclient: dump: 2023-10-30T20:23:37.208 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:37.208 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:37.209 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:37.209 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:37.209 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:37.209 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:37.209 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:37.210 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:37.210 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:37.210 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:37.210 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.192+0000 7f623440e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:37.210 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.192+0000 7f623440e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:37.211 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.192+0000 7f623440e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:37.211 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.192+0000 7f623440e640 1 -- 172.21.15.69:0/1671123431 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f622c04e9b0 con 0x7f622c0a9080 2023-10-30T20:23:37.425 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.423+0000 7f6222ffd640 1 -- 172.21.15.69:0/1671123431 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7f621c013470 con 0x7f622c0a9080 2023-10-30T20:23:37.425 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.424+0000 7f6222ffd640 1 --2- 172.21.15.69:0/1671123431 >> v2:172.21.15.69:6800/1876307353 conn(0x7f621006c730 0x7f621006ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:37.426 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.424+0000 7f62321aa640 1 --2- 172.21.15.69:0/1671123431 >> v2:172.21.15.69:6800/1876307353 conn(0x7f621006c730 0x7f621006ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:37.426 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.424+0000 7f62321aa640 10 monclient: get_auth_request con 0x7f621006c730 auth_method 0 2023-10-30T20:23:37.426 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.424+0000 7f62321aa640 1 --2- 172.21.15.69:0/1671123431 >> v2:172.21.15.69:6800/1876307353 conn(0x7f621006c730 0x7f621006ebf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6228036000 tx=0x7f622803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:37.427 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.424+0000 7f6222ffd640 10 client.4187.objecter ms_handle_connect 0x7f621006c730 2023-10-30T20:23:37.428 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.427+0000 7f6222ffd640 1 -- 172.21.15.69:0/1671123431 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f621c00ce60 con 0x7f622c0a9080 2023-10-30T20:23:37.428 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.427+0000 7f6222ffd640 10 client.4187.objecter ms_dispatch 0x7f622c000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T20:23:37.428 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.427+0000 7f6222ffd640 3 client.4187.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T20:23:37.429 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.427+0000 7f6222ffd640 3 client.4187.objecter handle_osd_map decoding full epoch 10 2023-10-30T20:23:37.429 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.427+0000 7f6222ffd640 20 client.4187.objecter dump_active .. 0 homeless 2023-10-30T20:23:37.637 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.636+0000 7f6222ffd640 1 -- 172.21.15.69:0/1671123431 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f621c0585b0 con 0x7f622c0a9080 2023-10-30T20:23:37.638 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.636+0000 7f6222ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:37.638 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.636+0000 7f6222ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:37.974 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.972+0000 7f623440e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:37.975 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.973+0000 7f623440e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:37.975 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.973+0000 7f623440e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:37.975 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.973+0000 7f623440e640 1 -- 172.21.15.69:0/1671123431 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f622c1d9db0 con 0x7f622c0a9080 2023-10-30T20:23:37.996 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.995+0000 7f6222ffd640 1 -- 172.21.15.69:0/1671123431 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f621c05c650 con 0x7f622c0a9080 2023-10-30T20:23:37.997 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.995+0000 7f6222ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:37.997 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.995+0000 7f6222ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:37.997 INFO:teuthology.misc.health.smithi069.stdout: 2023-10-30T20:23:37.997 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"32c1a723-7694-47d8-8b37-b1257dfe1796","created":"2023-10-30T20:23:06.559937+0000","modified":"2023-10-30T20:23:12.023084+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T20:23:12.023084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af10bc15-318a-4349-af76-32a30c8acf80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d0d32a9a-e24a-4d0c-9b90-125485d70672","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"508f2c3a-9e54-46aa-b6c0-52fad97fd341","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"003a2613-1050-4a96-b927-0e5983687319","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"678fc58e-ed6a-4fd0-9825-93297e3af6f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f945159a-a53c-48ba-97df-9374f9e56ffd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8184f59d-15e2-4bdc-90e3-fe1e231ca125","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T20:23:37.999 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.997+0000 7f623440e640 10 librados: watch_flush enter 2023-10-30T20:23:37.999 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.997+0000 7f623440e640 10 librados: watch_flush exit 2023-10-30T20:23:37.999 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.997+0000 7f623440e640 20 client.4187.objecter shutdown clearing up homeless session... 2023-10-30T20:23:38.001 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.997+0000 7f623440e640 10 client.4187.objecter successfully canceled tick 2023-10-30T20:23:38.001 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.998+0000 7f623440e640 1 -- 172.21.15.69:0/1671123431 >> v2:172.21.15.69:6800/1876307353 conn(0x7f621006c730 msgr2=0x7f621006ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:38.002 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.998+0000 7f623440e640 1 --2- 172.21.15.69:0/1671123431 >> v2:172.21.15.69:6800/1876307353 conn(0x7f621006c730 0x7f621006ebf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6228036000 tx=0x7f622803d040 comp rx=0 tx=0).stop 2023-10-30T20:23:38.002 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.998+0000 7f623440e640 10 monclient: shutdown 2023-10-30T20:23:38.002 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.998+0000 7f623440e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:38.002 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.998+0000 7f623440e640 1 -- 172.21.15.69:0/1671123431 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f622c0a9080 msgr2=0x7f622c0acca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:38.002 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.998+0000 7f623440e640 1 --2- 172.21.15.69:0/1671123431 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f622c0a9080 0x7f622c0acca0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f621c00e8a0 tx=0x7f621c00ed60 comp rx=0 tx=0).stop 2023-10-30T20:23:38.003 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.998+0000 7f623440e640 1 -- 172.21.15.69:0/1671123431 shutdown_connections 2023-10-30T20:23:38.003 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.998+0000 7f623440e640 1 --2- 172.21.15.69:0/1671123431 >> v2:172.21.15.69:6800/1876307353 conn(0x7f621006c730 0x7f621006ebf0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:38.003 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.998+0000 7f623440e640 1 --2- 172.21.15.69:0/1671123431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f622c0ad1e0 0x7f622c1363a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:38.003 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.998+0000 7f623440e640 1 --2- 172.21.15.69:0/1671123431 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f622c0a9080 0x7f622c0acca0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:38.003 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.998+0000 7f623440e640 1 --2- 172.21.15.69:0/1671123431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f622c0a8ca0 0x7f622c13b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:38.003 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.998+0000 7f623440e640 1 -- 172.21.15.69:0/1671123431 >> 172.21.15.69:0/1671123431 conn(0x7f622c0b71a0 msgr2=0x7f622c137a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:38.004 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.999+0000 7f623440e640 1 -- 172.21.15.69:0/1671123431 shutdown_connections 2023-10-30T20:23:38.004 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.999+0000 7f623440e640 1 -- 172.21.15.69:0/1671123431 wait complete. 2023-10-30T20:23:38.004 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:37.999+0000 7f623440e640 1 librados: shutdown 2023-10-30T20:23:38.013 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T20:23:39.028 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:26.521 86337== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:40.046 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:27.514 86340== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:40.140 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:27.621 86339== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:40.356 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:27.842 86403== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:40.533 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:28.032 86401== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:40.629 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:28.116 86338== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:40.763 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:28.254 86402== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:41.072 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:28.574 86399== Warning: unimplemented fcntl command: 1038 2023-10-30T20:23:44.819 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T20:23:44.959 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.958+0000 7fe2e34fc640 10 monclient: get_monmap_and_config 2023-10-30T20:23:44.960 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.958+0000 7fe2e34fc640 10 monclient: build_initial_monmap 2023-10-30T20:23:44.960 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.958+0000 7fe2e34fc640 10 monclient: monmap: 2023-10-30T20:23:44.960 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-10-30T20:23:44.960 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:44.960 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:23:44.959053+0000 2023-10-30T20:23:44.961 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:23:44.959053+0000 2023-10-30T20:23:44.961 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:44.961 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:44.961 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:44.961 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:44.962 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:44.962 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:44.962 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:44.962 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:44.965 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.958+0000 7fe2e34fc640 1 Processor -- start 2023-10-30T20:23:44.965 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.959+0000 7fe2e34fc640 1 -- start start 2023-10-30T20:23:44.965 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.959+0000 7fe2e34fc640 10 monclient: init 2023-10-30T20:23:44.966 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.960+0000 7fe2e34fc640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:44.966 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.960+0000 7fe2e34fc640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-10-30T20:23:44.966 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.960+0000 7fe2e34fc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe2dc0b4350 0x7fe2dc0aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:44.966 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.960+0000 7fe2e34fc640 10 monclient(hunting): picked mon.noname-c con 0x7fe2dc0b4350 addr v2:172.21.15.183:3300/0 2023-10-30T20:23:44.967 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.960+0000 7fe2e34fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2dc0adc70 addr v1:172.21.15.183:6789/0 2023-10-30T20:23:44.967 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.960+0000 7fe2e34fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe2dc13ebb0 0x7fe2dc13f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:44.967 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.960+0000 7fe2e34fc640 10 monclient(hunting): picked mon.noname-a con 0x7fe2dc13ebb0 addr v2:172.21.15.69:3300/0 2023-10-30T20:23:44.967 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.960+0000 7fe2e34fc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2dc0af3b0 con 0x7fe2dc0adc70 2023-10-30T20:23:44.967 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.960+0000 7fe2e34fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:44.968 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.960+0000 7fe2e34fc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2dc067a10 con 0x7fe2dc13ebb0 2023-10-30T20:23:44.968 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.960+0000 7fe2e34fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:44.968 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.960+0000 7fe2e34fc640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2dc13f580 con 0x7fe2dc0b4350 2023-10-30T20:23:44.968 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.960+0000 7fe2e34fc640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:44.969 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.960+0000 7fe2e34fc640 10 monclient(hunting): authenticate will time out at 1548.249111s 2023-10-30T20:23:44.969 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.961+0000 7fe2e1298640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe2dc0b4350 0x7fe2dc0aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:44.969 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.961+0000 7fe2e1a99640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe2dc13ebb0 0x7fe2dc13f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:44.969 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.961+0000 7fe2e1298640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe2dc0b4350 0x7fe2dc0aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:53040/0 (socket says 172.21.15.69:53040) 2023-10-30T20:23:44.969 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.961+0000 7fe2e1298640 1 -- 172.21.15.69:0/2174925649 learned_addr learned my addr 172.21.15.69:0/2174925649 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:23:44.970 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.961+0000 7fe2e1298640 10 monclient(hunting): get_auth_request con 0x7fe2dc0b4350 auth_method 0 2023-10-30T20:23:44.970 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.961+0000 7fe2e1298640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:44.970 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.961+0000 7fe2e1298640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:44.970 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.961+0000 7fe2e1298640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:44.970 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.961+0000 7fe2e1a99640 10 monclient(hunting): get_auth_request con 0x7fe2dc13ebb0 auth_method 0 2023-10-30T20:23:44.971 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.961+0000 7fe2e1a99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:44.971 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.961+0000 7fe2e1a99640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:44.971 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.961+0000 7fe2e1a99640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:44.971 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.962+0000 7fe2e1a99640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:44.972 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.962+0000 7fe2e1a99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:44.972 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.962+0000 7fe2e1a99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:44.972 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.962+0000 7fe2e1298640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:44.972 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.962+0000 7fe2e1298640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:44.972 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.962+0000 7fe2e1298640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:44.974 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.964+0000 7fe2e1a99640 10 monclient(hunting): handle_auth_done global_id 4209 payload 274 2023-10-30T20:23:44.974 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.964+0000 7fe2e1a99640 1 -- 172.21.15.69:0/2174925649 >> v2:172.21.15.183:3300/0 conn(0x7fe2dc0b4350 msgr2=0x7fe2dc0aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:44.974 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.964+0000 7fe2e1a99640 1 --2- 172.21.15.69:0/2174925649 >> v2:172.21.15.183:3300/0 conn(0x7fe2dc0b4350 0x7fe2dc0aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:44.974 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.965+0000 7fe2e1a99640 1 -- 172.21.15.69:0/2174925649 >> v1:172.21.15.183:6789/0 conn(0x7fe2dc0adc70 legacy=0x7fe2dc0b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:44.975 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.965+0000 7fe2e1a99640 10 monclient: _finish_hunting 0 2023-10-30T20:23:44.975 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.965+0000 7fe2e1a99640 1 monclient: found mon.noname-a 2023-10-30T20:23:44.975 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.965+0000 7fe2e1a99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:44.975 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.965+0000 7fe2e1a99640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:23:44.976 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.965+0000 7fe2e1a99640 1 -- 172.21.15.69:0/2174925649 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc0ae950 con 0x7fe2dc13ebb0 2023-10-30T20:23:44.976 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.967+0000 7fe2e1a99640 1 --2- 172.21.15.69:0/2174925649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc13ebb0 0x7fe2dc13f040 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0b3340 tx=0x7fe2d402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f130ccefb4ce9e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:44.979 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.973+0000 7fe2d3fff640 1 -- 172.21.15.69:0/2174925649 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d402cbf0 con 0x7fe2dc13ebb0 2023-10-30T20:23:44.979 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.973+0000 7fe2d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:44.980 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.973+0000 7fe2d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:23:44.980 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.973+0000 7fe2d3fff640 10 monclient: dump: 2023-10-30T20:23:44.980 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:44.980 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:44.980 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:44.981 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:44.981 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:44.981 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:44.981 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:44.981 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:44.982 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:44.982 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:44.982 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.974+0000 7fe2d3fff640 10 monclient: _finish_auth 0 2023-10-30T20:23:44.982 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.974+0000 7fe2d3fff640 10 monclient: _check_auth_tickets 2023-10-30T20:23:44.982 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.974+0000 7fe2d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:44.983 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.974+0000 7fe2d3fff640 1 -- 172.21.15.69:0/2174925649 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d402cd60 con 0x7fe2dc13ebb0 2023-10-30T20:23:44.983 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.974+0000 7fe2e34fc640 5 monclient: authenticate success, global_id 4209 2023-10-30T20:23:44.983 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.974+0000 7fe2e34fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:23:44.984 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.974+0000 7fe2d3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:44.984 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.974+0000 7fe2e34fc640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:44.984 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.974+0000 7fe2e34fc640 4 set_mon_vals no callback set 2023-10-30T20:23:44.984 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.974+0000 7fe2d3fff640 1 -- 172.21.15.69:0/2174925649 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4034660 con 0x7fe2dc13ebb0 2023-10-30T20:23:44.985 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.974+0000 7fe2d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:44.985 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.974+0000 7fe2d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:23:44.985 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.974+0000 7fe2d3fff640 10 monclient: dump: 2023-10-30T20:23:44.985 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:44.985 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:44.986 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:44.986 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:44.986 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:44.986 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:44.986 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:44.987 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:44.987 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:44.987 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:44.987 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.975+0000 7fe2e34fc640 10 monclient: shutdown 2023-10-30T20:23:44.987 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.975+0000 7fe2e34fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:44.988 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.975+0000 7fe2e34fc640 1 -- 172.21.15.69:0/2174925649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc13ebb0 msgr2=0x7fe2dc13f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:44.988 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.975+0000 7fe2e34fc640 1 --2- 172.21.15.69:0/2174925649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc13ebb0 0x7fe2dc13f040 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0b3340 tx=0x7fe2d402be40 comp rx=0 tx=0).stop 2023-10-30T20:23:44.988 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.975+0000 7fe2e34fc640 1 -- 172.21.15.69:0/2174925649 shutdown_connections 2023-10-30T20:23:44.988 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.975+0000 7fe2e34fc640 1 --2- 172.21.15.69:0/2174925649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc13ebb0 0x7fe2dc13f040 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:44.988 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.975+0000 7fe2e34fc640 1 --2- 172.21.15.69:0/2174925649 >> v2:172.21.15.183:3300/0 conn(0x7fe2dc0b4350 0x7fe2dc0aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:44.989 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.975+0000 7fe2e34fc640 1 -- 172.21.15.69:0/2174925649 >> 172.21.15.69:0/2174925649 conn(0x7fe2dc060bf0 msgr2=0x7fe2dc061000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:44.989 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.976+0000 7fe2e34fc640 1 -- 172.21.15.69:0/2174925649 shutdown_connections 2023-10-30T20:23:44.989 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.976+0000 7fe2e34fc640 1 -- 172.21.15.69:0/2174925649 wait complete. 2023-10-30T20:23:44.989 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.977+0000 7fe2e34fc640 10 monclient: build_initial_monmap 2023-10-30T20:23:44.989 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.977+0000 7fe2e34fc640 10 monclient: monmap: 2023-10-30T20:23:44.990 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-10-30T20:23:44.990 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:44.990 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2023-10-30T20:23:44.990 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2023-10-30T20:23:44.990 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:44.991 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-10-30T20:23:44.991 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:44.991 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:44.991 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:44.991 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:44.992 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.977+0000 7fe2e34fc640 1 librados: starting msgr at 2023-10-30T20:23:44.992 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.977+0000 7fe2e34fc640 1 librados: starting objecter 2023-10-30T20:23:44.992 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.977+0000 7fe2e34fc640 1 Processor -- start 2023-10-30T20:23:44.992 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.977+0000 7fe2e34fc640 1 -- start start 2023-10-30T20:23:44.992 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.977+0000 7fe2e34fc640 1 librados: setting wanted keys 2023-10-30T20:23:44.993 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.977+0000 7fe2e34fc640 1 librados: calling monclient init 2023-10-30T20:23:44.993 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.977+0000 7fe2e34fc640 10 monclient: init 2023-10-30T20:23:44.994 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.978+0000 7fe2e34fc640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:44.994 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.978+0000 7fe2e34fc640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:23:44.994 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.978+0000 7fe2e34fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0adc70 0x7fe2dc0a9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:44.995 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.979+0000 7fe2e34fc640 10 monclient(hunting): picked mon.noname-a con 0x7fe2dc0adc70 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:44.995 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.979+0000 7fe2e34fc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2dc0b4350 0x7fe2dc1da250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:44.995 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.979+0000 7fe2e34fc640 10 monclient(hunting): picked mon.noname-b con 0x7fe2dc0b4350 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:44.995 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.979+0000 7fe2e34fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc13ebb0 0x7fe2dc139360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:44.995 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.979+0000 7fe2e1298640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0adc70 0x7fe2dc0a9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:44.996 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.979+0000 7fe2e1298640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0adc70 0x7fe2dc0a9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35892/0 (socket says 172.21.15.69:35892) 2023-10-30T20:23:44.996 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.979+0000 7fe2e1298640 1 -- 172.21.15.69:0/1065303901 learned_addr learned my addr 172.21.15.69:0/1065303901 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:23:44.996 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.979+0000 7fe2e34fc640 10 monclient(hunting): picked mon.noname-c con 0x7fe2dc13ebb0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:44.996 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.979+0000 7fe2e34fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:44.996 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.979+0000 7fe2e34fc640 1 -- 172.21.15.69:0/1065303901 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0b3630 con 0x7fe2dc0adc70 2023-10-30T20:23:44.997 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e34fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:44.997 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e34fc640 1 -- 172.21.15.69:0/1065303901 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc1d9dc0 con 0x7fe2dc0b4350 2023-10-30T20:23:44.997 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e34fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:44.997 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e34fc640 1 -- 172.21.15.69:0/1065303901 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc1d9a00 con 0x7fe2dc13ebb0 2023-10-30T20:23:44.997 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e34fc640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:44.998 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e34fc640 10 monclient(hunting): authenticate will time out at 1548.268840s 2023-10-30T20:23:44.998 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e1298640 10 monclient(hunting): get_auth_request con 0x7fe2dc0adc70 auth_method 0 2023-10-30T20:23:44.998 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e1298640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:44.998 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e1298640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:44.998 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e1298640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:44.999 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e1a99640 1 --2- 172.21.15.69:0/1065303901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc13ebb0 0x7fe2dc139360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:44.999 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e1a99640 10 monclient(hunting): get_auth_request con 0x7fe2dc13ebb0 auth_method 0 2023-10-30T20:23:44.999 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e1a99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:44.999 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e1a99640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:44.999 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e1a99640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:45.000 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e1298640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:45.000 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e1298640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:45.000 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.980+0000 7fe2e1298640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:45.000 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.981+0000 7fe2e1a99640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:45.000 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.981+0000 7fe2e1a99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:45.001 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.981+0000 7fe2e1a99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:45.001 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.984+0000 7fe2e1298640 10 monclient(hunting): handle_auth_done global_id 4212 payload 824 2023-10-30T20:23:45.001 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.984+0000 7fe2e1298640 1 -- 172.21.15.69:0/1065303901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc13ebb0 msgr2=0x7fe2dc139360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:45.001 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.984+0000 7fe2e1298640 1 --2- 172.21.15.69:0/1065303901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc13ebb0 0x7fe2dc139360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:45.001 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.984+0000 7fe2e1298640 1 -- 172.21.15.69:0/1065303901 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2dc0b4350 msgr2=0x7fe2dc1da250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:45.002 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.984+0000 7fe2e1298640 1 --2- 172.21.15.69:0/1065303901 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2dc0b4350 0x7fe2dc1da250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:45.002 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.984+0000 7fe2e1298640 10 monclient: _finish_hunting 0 2023-10-30T20:23:45.002 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.984+0000 7fe2e1298640 1 monclient: found mon.noname-a 2023-10-30T20:23:45.002 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.984+0000 7fe2e1298640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:45.002 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.984+0000 7fe2e1298640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:23:45.003 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.984+0000 7fe2e1298640 1 -- 172.21.15.69:0/1065303901 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc1da790 con 0x7fe2dc0adc70 2023-10-30T20:23:45.003 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.988+0000 7fe2e1298640 1 --2- 172.21.15.69:0/1065303901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0adc70 0x7fe2dc0a9420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8009fd0 tx=0x7fe2d800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:45.003 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.988+0000 7fe2d1ffb640 10 client.?.objecter ms_handle_connect 0x7fe2dc0adc70 2023-10-30T20:23:45.003 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.988+0000 7fe2d1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:45.005 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2d1ffb640 1 -- 172.21.15.69:0/1065303901 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d800c8e0 con 0x7fe2dc0adc70 2023-10-30T20:23:45.005 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:45.005 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2d1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:23:45.006 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2d1ffb640 10 monclient: dump: 2023-10-30T20:23:45.006 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:45.006 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:45.006 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:45.007 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:45.007 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:45.007 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:45.007 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:45.007 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:45.007 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:45.008 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:45.008 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2d1ffb640 10 monclient: _finish_auth 0 2023-10-30T20:23:45.008 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2d1ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:23:45.008 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:45.008 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2d1ffb640 1 -- 172.21.15.69:0/1065303901 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d800d040 con 0x7fe2dc0adc70 2023-10-30T20:23:45.009 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2e34fc640 5 monclient: authenticate success, global_id 4212 2023-10-30T20:23:45.009 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2e34fc640 10 monclient: _renew_subs 2023-10-30T20:23:45.009 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2e34fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:45.009 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2e34fc640 1 -- 172.21.15.69:0/1065303901 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2dc1398a0 con 0x7fe2dc0adc70 2023-10-30T20:23:45.009 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2d1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:45.010 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2d1ffb640 1 -- 172.21.15.69:0/1065303901 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d800cc50 con 0x7fe2dc0adc70 2023-10-30T20:23:45.010 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:45.010 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:23:45.010 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2d1ffb640 10 monclient: dump: 2023-10-30T20:23:45.010 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:45.011 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:45.011 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:45.011 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:45.011 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:45.011 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:45.012 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:45.012 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:45.012 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:45.012 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:45.012 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2d37fe640 4 set_mon_vals no callback set 2023-10-30T20:23:45.013 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2e34fc640 10 client.4212.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:45.013 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2e34fc640 10 monclient: _renew_subs 2023-10-30T20:23:45.013 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2e34fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:45.013 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2e34fc640 1 -- 172.21.15.69:0/1065303901 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc139d60 con 0x7fe2dc0adc70 2023-10-30T20:23:45.013 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.994+0000 7fe2e34fc640 1 librados: init done 2023-10-30T20:23:45.014 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.996+0000 7fe2e34fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:45.014 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.996+0000 7fe2e34fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:45.014 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.996+0000 7fe2e34fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:45.014 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:44.996+0000 7fe2e34fc640 1 -- 172.21.15.69:0/1065303901 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2dc04e9b0 con 0x7fe2dc0adc70 2023-10-30T20:23:45.289 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.286+0000 7fe2d1ffb640 1 -- 172.21.15.69:0/1065303901 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7fe2d8024480 con 0x7fe2dc0adc70 2023-10-30T20:23:45.289 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.287+0000 7fe2d1ffb640 1 --2- 172.21.15.69:0/1065303901 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe2c006c760 0x7fe2c006ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:45.289 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.287+0000 7fe2e0a97640 1 --2- 172.21.15.69:0/1065303901 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe2c006c760 0x7fe2c006ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:45.290 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.287+0000 7fe2e0a97640 10 monclient: get_auth_request con 0x7fe2c006c760 auth_method 0 2023-10-30T20:23:45.290 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.288+0000 7fe2e0a97640 1 --2- 172.21.15.69:0/1065303901 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe2c006c760 0x7fe2c006ec20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc000f60 tx=0x7fe2cc0079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:45.290 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.288+0000 7fe2d1ffb640 10 client.4212.objecter ms_handle_connect 0x7fe2c006c760 2023-10-30T20:23:45.362 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.360+0000 7fe2d1ffb640 1 -- 172.21.15.69:0/1065303901 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fe2d8069670 con 0x7fe2dc0adc70 2023-10-30T20:23:45.362 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.360+0000 7fe2d1ffb640 10 client.4212.objecter ms_dispatch 0x7fe2dc000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T20:23:45.362 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.360+0000 7fe2d1ffb640 3 client.4212.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T20:23:45.363 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.360+0000 7fe2d1ffb640 3 client.4212.objecter handle_osd_map decoding full epoch 10 2023-10-30T20:23:45.363 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.360+0000 7fe2d1ffb640 20 client.4212.objecter dump_active .. 0 homeless 2023-10-30T20:23:45.444 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.442+0000 7fe2d1ffb640 1 -- 172.21.15.69:0/1065303901 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe2d806d710 con 0x7fe2dc0adc70 2023-10-30T20:23:45.444 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.443+0000 7fe2d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:45.444 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.443+0000 7fe2d1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:45.667 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.665+0000 7fe2e34fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:45.667 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.665+0000 7fe2e34fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:45.668 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.665+0000 7fe2e34fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:23:45.668 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.665+0000 7fe2e34fc640 1 -- 172.21.15.69:0/1065303901 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2dc139d60 con 0x7fe2dc0adc70 2023-10-30T20:23:45.739 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.738+0000 7fe2d1ffb640 1 -- 172.21.15.69:0/1065303901 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe2d80718c0 con 0x7fe2dc0adc70 2023-10-30T20:23:45.740 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.738+0000 7fe2d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:45.740 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.738+0000 7fe2d1ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:45.740 INFO:teuthology.misc.health.smithi069.stdout: 2023-10-30T20:23:45.741 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"32c1a723-7694-47d8-8b37-b1257dfe1796","created":"2023-10-30T20:23:06.559937+0000","modified":"2023-10-30T20:23:12.023084+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T20:23:12.023084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af10bc15-318a-4349-af76-32a30c8acf80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d0d32a9a-e24a-4d0c-9b90-125485d70672","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"508f2c3a-9e54-46aa-b6c0-52fad97fd341","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"003a2613-1050-4a96-b927-0e5983687319","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"678fc58e-ed6a-4fd0-9825-93297e3af6f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f945159a-a53c-48ba-97df-9374f9e56ffd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8184f59d-15e2-4bdc-90e3-fe1e231ca125","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T20:23:45.742 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.741+0000 7fe2e34fc640 10 librados: watch_flush enter 2023-10-30T20:23:45.742 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.741+0000 7fe2e34fc640 10 librados: watch_flush exit 2023-10-30T20:23:45.743 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.741+0000 7fe2e34fc640 20 client.4212.objecter shutdown clearing up homeless session... 2023-10-30T20:23:45.743 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.741+0000 7fe2e34fc640 10 client.4212.objecter successfully canceled tick 2023-10-30T20:23:45.744 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.741+0000 7fe2e34fc640 1 -- 172.21.15.69:0/1065303901 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe2c006c760 msgr2=0x7fe2c006ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:45.744 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.741+0000 7fe2e34fc640 1 --2- 172.21.15.69:0/1065303901 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe2c006c760 0x7fe2c006ec20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc000f60 tx=0x7fe2cc0079b0 comp rx=0 tx=0).stop 2023-10-30T20:23:45.744 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.741+0000 7fe2e34fc640 10 monclient: shutdown 2023-10-30T20:23:45.744 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.741+0000 7fe2e34fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:45.745 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.741+0000 7fe2e34fc640 1 -- 172.21.15.69:0/1065303901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0adc70 msgr2=0x7fe2dc0a9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:45.745 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.741+0000 7fe2e34fc640 1 --2- 172.21.15.69:0/1065303901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0adc70 0x7fe2dc0a9420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8009fd0 tx=0x7fe2d800ed40 comp rx=0 tx=0).stop 2023-10-30T20:23:45.745 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.742+0000 7fe2e34fc640 1 -- 172.21.15.69:0/1065303901 shutdown_connections 2023-10-30T20:23:45.745 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.742+0000 7fe2e34fc640 1 --2- 172.21.15.69:0/1065303901 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe2c006c760 0x7fe2c006ec20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:45.745 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.742+0000 7fe2e34fc640 1 --2- 172.21.15.69:0/1065303901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc13ebb0 0x7fe2dc139360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:45.745 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.742+0000 7fe2e34fc640 1 --2- 172.21.15.69:0/1065303901 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2dc0b4350 0x7fe2dc1da250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:45.746 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.742+0000 7fe2e34fc640 1 --2- 172.21.15.69:0/1065303901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0adc70 0x7fe2dc0a9420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:45.746 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.742+0000 7fe2e34fc640 1 -- 172.21.15.69:0/1065303901 >> 172.21.15.69:0/1065303901 conn(0x7fe2dc060bf0 msgr2=0x7fe2dc13b040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:45.746 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.742+0000 7fe2e34fc640 1 -- 172.21.15.69:0/1065303901 shutdown_connections 2023-10-30T20:23:45.747 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.742+0000 7fe2e34fc640 1 -- 172.21.15.69:0/1065303901 wait complete. 2023-10-30T20:23:45.747 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:45.742+0000 7fe2e34fc640 1 librados: shutdown 2023-10-30T20:23:45.757 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T20:23:46.912 INFO:tasks.ceph.osd.4.smithi183.stderr:2023-10-30T20:23:46.908+0000 5dc2340 -1 osd.4 0 log_to_monitors true 2023-10-30T20:23:48.279 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:23:48.275+0000 5dc2340 -1 osd.7 0 log_to_monitors true 2023-10-30T20:23:48.620 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-10-30T20:23:48.614+0000 5dc2340 -1 osd.2 0 log_to_monitors true 2023-10-30T20:23:48.800 INFO:tasks.ceph.osd.6.smithi183.stderr:2023-10-30T20:23:48.741+0000 5dc2340 -1 osd.6 0 log_to_monitors true 2023-10-30T20:23:49.329 INFO:tasks.ceph.osd.5.smithi183.stderr:2023-10-30T20:23:49.251+0000 5dc2340 -1 osd.5 0 log_to_monitors true 2023-10-30T20:23:49.384 INFO:tasks.ceph.osd.3.smithi069.stderr:2023-10-30T20:23:49.380+0000 5dc2340 -1 osd.3 0 log_to_monitors true 2023-10-30T20:23:49.436 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-10-30T20:23:49.432+0000 5dc2340 -1 osd.0 0 log_to_monitors true 2023-10-30T20:23:49.492 INFO:tasks.ceph.osd.1.smithi069.stderr:2023-10-30T20:23:49.486+0000 5dc2340 -1 osd.1 0 log_to_monitors true 2023-10-30T20:23:50.451 INFO:tasks.ceph.osd.4.smithi183.stderr:2023-10-30T20:23:50.439+0000 b47e640 -1 osd.4 0 waiting for initial osdmap 2023-10-30T20:23:50.830 INFO:tasks.ceph.osd.4.smithi183.stderr:2023-10-30T20:23:50.817+0000 1025a640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T20:23:51.745 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-10-30T20:23:51.741+0000 b47e640 -1 osd.2 0 waiting for initial osdmap 2023-10-30T20:23:51.768 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:23:51.763+0000 bc94640 -1 osd.7 0 waiting for initial osdmap 2023-10-30T20:23:51.847 INFO:tasks.ceph.osd.6.smithi183.stderr:2023-10-30T20:23:51.841+0000 bc94640 -1 osd.6 0 waiting for initial osdmap 2023-10-30T20:23:52.145 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-10-30T20:23:52.131+0000 11a5d640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T20:23:52.180 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:23:52.165+0000 1125c640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T20:23:52.270 INFO:tasks.ceph.osd.6.smithi183.stderr:2023-10-30T20:23:52.256+0000 10a5b640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T20:23:52.564 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T20:23:52.665 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.663+0000 7ff7b624f640 10 monclient: get_monmap_and_config 2023-10-30T20:23:52.665 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.663+0000 7ff7b624f640 10 monclient: build_initial_monmap 2023-10-30T20:23:52.665 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.663+0000 7ff7b624f640 10 monclient: monmap: 2023-10-30T20:23:52.665 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-10-30T20:23:52.666 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:52.666 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:23:52.664616+0000 2023-10-30T20:23:52.666 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:23:52.664616+0000 2023-10-30T20:23:52.666 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:52.667 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:52.667 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:23:52.667 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:23:52.667 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:23:52.667 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:23:52.667 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:23:52.668 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:52.670 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.664+0000 7ff7b624f640 1 Processor -- start 2023-10-30T20:23:52.670 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.664+0000 7ff7b624f640 1 -- start start 2023-10-30T20:23:52.670 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.664+0000 7ff7b624f640 10 monclient: init 2023-10-30T20:23:52.671 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.665+0000 7ff7b624f640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:52.671 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.665+0000 7ff7b624f640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-10-30T20:23:52.671 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.665+0000 7ff7b624f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b00b4350 0x7ff7b00aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:52.671 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.665+0000 7ff7b624f640 10 monclient(hunting): picked mon.noname-b con 0x7ff7b00b4350 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:52.671 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.665+0000 7ff7b624f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff7b00adc70 0x7ff7b00b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:52.672 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.665+0000 7ff7b624f640 10 monclient(hunting): picked mon.noname-a con 0x7ff7b00adc70 addr v2:172.21.15.69:3300/0 2023-10-30T20:23:52.672 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.665+0000 7ff7b624f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff7b00b0d50 addr v1:172.21.15.183:6789/0 2023-10-30T20:23:52.672 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.665+0000 7ff7b624f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b00af3b0 con 0x7ff7b00b0d50 2023-10-30T20:23:52.672 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.665+0000 7ff7b624f640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:52.672 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.665+0000 7ff7b624f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b0067a10 con 0x7ff7b00adc70 2023-10-30T20:23:52.673 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.665+0000 7ff7b624f640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:52.673 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.665+0000 7ff7b624f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b00b4730 con 0x7ff7b00b4350 2023-10-30T20:23:52.673 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.665+0000 7ff7b624f640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:52.673 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.665+0000 7ff7b624f640 10 monclient(hunting): authenticate will time out at 1555.954179s 2023-10-30T20:23:52.673 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.666+0000 7ff7af7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b00b4350 0x7ff7b00aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:52.674 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.666+0000 7ff7af7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b00b4350 0x7ff7b00aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34252/0 (socket says 172.21.15.69:34252) 2023-10-30T20:23:52.674 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.666+0000 7ff7af7fe640 1 -- 172.21.15.69:0/2729770596 learned_addr learned my addr 172.21.15.69:0/2729770596 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:23:52.674 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.666+0000 7ff7af7fe640 10 monclient(hunting): get_auth_request con 0x7ff7b00b4350 auth_method 0 2023-10-30T20:23:52.674 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.666+0000 7ff7af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:52.674 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.666+0000 7ff7af7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:52.675 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.666+0000 7ff7af7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:52.675 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.667+0000 7ff7af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:52.675 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.667+0000 7ff7af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:52.675 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.667+0000 7ff7af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:52.675 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.668+0000 7ff7af7fe640 10 monclient(hunting): handle_auth_done global_id 4214 payload 274 2023-10-30T20:23:52.676 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.668+0000 7ff7af7fe640 1 -- 172.21.15.69:0/2729770596 >> v2:172.21.15.69:3300/0 conn(0x7ff7b00adc70 msgr2=0x7ff7b00b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:52.676 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.668+0000 7ff7af7fe640 1 --2- 172.21.15.69:0/2729770596 >> v2:172.21.15.69:3300/0 conn(0x7ff7b00adc70 0x7ff7b00b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:52.676 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.668+0000 7ff7af7fe640 1 -- 172.21.15.69:0/2729770596 >> v1:172.21.15.183:6789/0 conn(0x7ff7b00b0d50 legacy=0x7ff7b013ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:23:52.676 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.668+0000 7ff7af7fe640 10 monclient: _finish_hunting 0 2023-10-30T20:23:52.676 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.668+0000 7ff7af7fe640 1 monclient: found mon.noname-b 2023-10-30T20:23:52.677 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.668+0000 7ff7af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:52.677 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.668+0000 7ff7af7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:23:52.677 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.668+0000 7ff7af7fe640 1 -- 172.21.15.69:0/2729770596 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b00ae950 con 0x7ff7b00b4350 2023-10-30T20:23:52.678 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.670+0000 7ff7af7fe640 1 --2- 172.21.15.69:0/2729770596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b00b4350 0x7ff7b00aee70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0009870 tx=0x7ff7a002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3326495b609bbeb7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:52.678 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.676+0000 7ff7ae7fc640 1 -- 172.21.15.69:0/2729770596 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00317c0 con 0x7ff7b00b4350 2023-10-30T20:23:52.678 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.676+0000 7ff7ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:52.679 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.676+0000 7ff7ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:23:52.679 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.676+0000 7ff7ae7fc640 10 monclient: dump: 2023-10-30T20:23:52.679 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:52.679 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:52.679 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:52.680 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:52.680 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:52.680 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:52.680 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:52.680 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:52.681 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:52.681 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:52.681 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.676+0000 7ff7ae7fc640 10 monclient: _finish_auth 0 2023-10-30T20:23:52.681 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.676+0000 7ff7ae7fc640 10 monclient: _check_auth_tickets 2023-10-30T20:23:52.681 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.676+0000 7ff7ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:52.682 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.676+0000 7ff7b624f640 5 monclient: authenticate success, global_id 4214 2023-10-30T20:23:52.682 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.676+0000 7ff7b624f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:23:52.682 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.676+0000 7ff7ae7fc640 1 -- 172.21.15.69:0/2729770596 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a002e070 con 0x7ff7b00b4350 2023-10-30T20:23:52.682 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.676+0000 7ff7ae7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:52.682 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.676+0000 7ff7b624f640 10 monclient: get_monmap_and_config success 2023-10-30T20:23:52.682 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.676+0000 7ff7b624f640 4 set_mon_vals no callback set 2023-10-30T20:23:52.685 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.677+0000 7ff7b624f640 10 monclient: shutdown 2023-10-30T20:23:52.686 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.677+0000 7ff7b624f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:52.686 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.677+0000 7ff7b624f640 1 -- 172.21.15.69:0/2729770596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b00b4350 msgr2=0x7ff7b00aee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:52.686 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.677+0000 7ff7ae7fc640 1 -- 172.21.15.69:0/2729770596 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00317c0 con 0x7ff7b00b4350 2023-10-30T20:23:52.686 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.677+0000 7ff7b624f640 1 --2- 172.21.15.69:0/2729770596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b00b4350 0x7ff7b00aee70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0009870 tx=0x7ff7a002b860 comp rx=0 tx=0).stop 2023-10-30T20:23:52.686 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.677+0000 7ff7ae7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2023-10-30T20:23:52.687 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.677+0000 7ff7b624f640 1 -- 172.21.15.69:0/2729770596 shutdown_connections 2023-10-30T20:23:52.687 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.677+0000 7ff7b624f640 1 --2- 172.21.15.69:0/2729770596 >> v2:172.21.15.69:3300/0 conn(0x7ff7b00adc70 0x7ff7b00b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:52.687 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.677+0000 7ff7b624f640 1 --2- 172.21.15.69:0/2729770596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b00b4350 0x7ff7b00aee70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:52.687 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.677+0000 7ff7b624f640 1 -- 172.21.15.69:0/2729770596 >> 172.21.15.69:0/2729770596 conn(0x7ff7b0060bf0 msgr2=0x7ff7b0061000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:52.687 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.678+0000 7ff7b624f640 1 -- 172.21.15.69:0/2729770596 shutdown_connections 2023-10-30T20:23:52.688 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.678+0000 7ff7b624f640 1 -- 172.21.15.69:0/2729770596 wait complete. 2023-10-30T20:23:52.688 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.678+0000 7ff7b624f640 10 monclient: build_initial_monmap 2023-10-30T20:23:52.688 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.679+0000 7ff7b624f640 10 monclient: monmap: 2023-10-30T20:23:52.688 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-10-30T20:23:52.688 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:23:52.689 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2023-10-30T20:23:52.689 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2023-10-30T20:23:52.689 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:23:52.689 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-10-30T20:23:52.689 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:23:52.690 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:23:52.690 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:23:52.690 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:52.690 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.679+0000 7ff7b624f640 1 librados: starting msgr at 2023-10-30T20:23:52.690 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.679+0000 7ff7b624f640 1 librados: starting objecter 2023-10-30T20:23:52.690 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.679+0000 7ff7b624f640 1 Processor -- start 2023-10-30T20:23:52.691 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.679+0000 7ff7b624f640 1 -- start start 2023-10-30T20:23:52.691 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.679+0000 7ff7b624f640 1 librados: setting wanted keys 2023-10-30T20:23:52.691 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.679+0000 7ff7b624f640 1 librados: calling monclient init 2023-10-30T20:23:52.691 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.679+0000 7ff7b624f640 10 monclient: init 2023-10-30T20:23:52.692 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 10 monclient: _reopen_session rank -1 2023-10-30T20:23:52.692 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:23:52.692 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b00adc70 0x7ff7b0135ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:52.692 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 10 monclient(hunting): picked mon.noname-a con 0x7ff7b00adc70 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:23:52.692 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b00b0d50 0x7ff7b0140820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:52.692 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 10 monclient(hunting): picked mon.noname-b con 0x7ff7b00b0d50 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:23:52.693 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b00b4350 0x7ff7b01da2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:52.693 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 10 monclient(hunting): picked mon.noname-c con 0x7ff7b00b4350 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:23:52.693 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:52.693 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b00b1130 con 0x7ff7b00adc70 2023-10-30T20:23:52.693 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:52.694 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b00aaf60 con 0x7ff7b00b0d50 2023-10-30T20:23:52.694 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 10 monclient(hunting): start opening mon connection 2023-10-30T20:23:52.694 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b00a9b20 con 0x7ff7b00b4350 2023-10-30T20:23:52.694 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 10 monclient(hunting): _renew_subs 2023-10-30T20:23:52.694 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.680+0000 7ff7b624f640 10 monclient(hunting): authenticate will time out at 1555.969762s 2023-10-30T20:23:52.695 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.681+0000 7ff7affff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b00b4350 0x7ff7b01da2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:52.695 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.681+0000 7ff7affff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b00b4350 0x7ff7b01da2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34268/0 (socket says 172.21.15.69:34268) 2023-10-30T20:23:52.695 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.681+0000 7ff7affff640 1 -- 172.21.15.69:0/949814664 learned_addr learned my addr 172.21.15.69:0/949814664 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:23:52.695 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.681+0000 7ff7affff640 10 monclient(hunting): get_auth_request con 0x7ff7b00b4350 auth_method 0 2023-10-30T20:23:52.695 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.681+0000 7ff7affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:23:52.696 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.681+0000 7ff7affff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:23:52.696 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.681+0000 7ff7affff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:23:52.696 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.682+0000 7ff7affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:23:52.696 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.682+0000 7ff7affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:23:52.696 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.682+0000 7ff7affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:23:52.700 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.685+0000 7ff7af7fe640 1 --2- 172.21.15.69:0/949814664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b00adc70 0x7ff7b0135ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:52.700 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.686+0000 7ff7affff640 10 monclient(hunting): handle_auth_done global_id 4217 payload 824 2023-10-30T20:23:52.700 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.686+0000 7ff7affff640 1 -- 172.21.15.69:0/949814664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b00b0d50 msgr2=0x7ff7b0140820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:52.701 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.686+0000 7ff7affff640 1 --2- 172.21.15.69:0/949814664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b00b0d50 0x7ff7b0140820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:52.701 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.686+0000 7ff7affff640 1 -- 172.21.15.69:0/949814664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b00adc70 msgr2=0x7ff7b0135ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:52.701 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.686+0000 7ff7affff640 1 --2- 172.21.15.69:0/949814664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b00adc70 0x7ff7b0135ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:52.701 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.686+0000 7ff7affff640 10 monclient: _finish_hunting 0 2023-10-30T20:23:52.702 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.686+0000 7ff7affff640 1 monclient: found mon.noname-c 2023-10-30T20:23:52.702 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.686+0000 7ff7affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:23:52.702 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.686+0000 7ff7affff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:23:52.702 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.686+0000 7ff7affff640 1 -- 172.21.15.69:0/949814664 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b01423d0 con 0x7ff7b00b4350 2023-10-30T20:23:52.702 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.690+0000 7ff7affff640 1 --2- 172.21.15.69:0/949814664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b00b4350 0x7ff7b01da2d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff7a800c780 tx=0x7ff7a800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:52.703 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.690+0000 7ff797fff640 10 client.?.objecter ms_handle_connect 0x7ff7b00b4350 2023-10-30T20:23:52.703 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.690+0000 7ff797fff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:23:52.703 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff797fff640 1 -- 172.21.15.69:0/949814664 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a800ce90 con 0x7ff7b00b4350 2023-10-30T20:23:52.703 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff797fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:52.703 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff797fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:23:52.704 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff797fff640 10 monclient: dump: 2023-10-30T20:23:52.704 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:52.704 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:52.704 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:52.704 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:52.705 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:52.705 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:52.705 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:52.705 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:52.705 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:52.705 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:52.706 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff797fff640 10 monclient: _finish_auth 0 2023-10-30T20:23:52.706 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff797fff640 10 monclient: _check_auth_tickets 2023-10-30T20:23:52.706 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff797fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:23:52.706 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff797fff640 1 -- 172.21.15.69:0/949814664 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a800e040 con 0x7ff7b00b4350 2023-10-30T20:23:52.706 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff797fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:23:52.707 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff797fff640 1 -- 172.21.15.69:0/949814664 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8005070 con 0x7ff7b00b4350 2023-10-30T20:23:52.707 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff797fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:23:52.707 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff797fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:23:52.707 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff797fff640 10 monclient: dump: 2023-10-30T20:23:52.707 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:23:52.708 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:23:52.708 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:52.708 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:23:52.708 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:23:52.708 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:23:52.709 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:23:52.709 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:23:52.709 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:23:52.709 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:23:52.709 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff7b624f640 5 monclient: authenticate success, global_id 4217 2023-10-30T20:23:52.710 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff7adffb640 4 set_mon_vals no callback set 2023-10-30T20:23:52.710 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff7b624f640 10 monclient: _renew_subs 2023-10-30T20:23:52.710 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff7b624f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:52.710 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff7b624f640 1 -- 172.21.15.69:0/949814664 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b01da810 con 0x7ff7b00b4350 2023-10-30T20:23:52.710 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff7b624f640 10 client.4217.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:23:52.710 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff7b624f640 10 monclient: _renew_subs 2023-10-30T20:23:52.711 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff7b624f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:52.711 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff7b624f640 1 -- 172.21.15.69:0/949814664 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b01dace0 con 0x7ff7b00b4350 2023-10-30T20:23:52.711 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.695+0000 7ff7b624f640 1 librados: init done 2023-10-30T20:23:52.711 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.696+0000 7ff7b624f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:52.712 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.696+0000 7ff7b624f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:52.712 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.696+0000 7ff7b624f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:52.712 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.696+0000 7ff7b624f640 1 -- 172.21.15.69:0/949814664 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b004e9b0 con 0x7ff7b00b4350 2023-10-30T20:23:52.736 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.735+0000 7ff797fff640 1 -- 172.21.15.69:0/949814664 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7ff7a80051e0 con 0x7ff7b00b4350 2023-10-30T20:23:52.737 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.736+0000 7ff797fff640 1 --2- 172.21.15.69:0/949814664 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff79c06c780 0x7ff79c06ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:23:52.737 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.736+0000 7ff7af7fe640 1 --2- 172.21.15.69:0/949814664 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff79c06c780 0x7ff79c06ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:23:52.738 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.736+0000 7ff7af7fe640 10 monclient: get_auth_request con 0x7ff79c06c780 auth_method 0 2023-10-30T20:23:52.738 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.736+0000 7ff797fff640 10 client.4217.objecter ms_handle_connect 0x7ff79c06c780 2023-10-30T20:23:52.738 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:52.736+0000 7ff7af7fe640 1 --2- 172.21.15.69:0/949814664 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff79c06c780 0x7ff79c06ec40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0036000 tx=0x7ff7a0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:23:52.843 INFO:tasks.ceph.osd.5.smithi183.stderr:2023-10-30T20:23:52.837+0000 b47e640 -1 osd.5 0 waiting for initial osdmap 2023-10-30T20:23:52.918 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-10-30T20:23:52.914+0000 b47e640 -1 osd.0 0 waiting for initial osdmap 2023-10-30T20:23:52.928 INFO:tasks.ceph.osd.3.smithi069.stderr:2023-10-30T20:23:52.918+0000 b47e640 -1 osd.3 0 waiting for initial osdmap 2023-10-30T20:23:53.066 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.064+0000 7ff797fff640 1 -- 172.21.15.69:0/949814664 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2651+0+0 (secure 0 0 0) 0x7ff7a8011e60 con 0x7ff7b00b4350 2023-10-30T20:23:53.066 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.064+0000 7ff797fff640 10 client.4217.objecter ms_dispatch 0x7ff7b0000b90 osd_map(14..14 src has 1..14) v4 2023-10-30T20:23:53.066 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.064+0000 7ff797fff640 3 client.4217.objecter handle_osd_map got epochs [14,14] > 0 2023-10-30T20:23:53.066 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.064+0000 7ff797fff640 3 client.4217.objecter handle_osd_map decoding full epoch 14 2023-10-30T20:23:53.067 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.064+0000 7ff797fff640 20 client.4217.objecter dump_active .. 0 homeless 2023-10-30T20:23:53.067 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.064+0000 7ff797fff640 1 -- 172.21.15.69:0/949814664 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff7a8058160 con 0x7ff7b00b4350 2023-10-30T20:23:53.067 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.064+0000 7ff797fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:23:53.067 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.064+0000 7ff797fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:23:53.247 INFO:tasks.ceph.osd.5.smithi183.stderr:2023-10-30T20:23:53.233+0000 1125c640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T20:23:53.254 INFO:tasks.ceph.osd.1.smithi069.stderr:2023-10-30T20:23:53.249+0000 bc94640 -1 osd.1 0 waiting for initial osdmap 2023-10-30T20:23:53.292 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.291+0000 7ff7b624f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:53.292 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.291+0000 7ff7b624f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:53.292 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.291+0000 7ff7b624f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:23:53.293 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.291+0000 7ff7b624f640 1 -- 172.21.15.69:0/949814664 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7b01dace0 con 0x7ff7b00b4350 2023-10-30T20:23:53.312 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.310+0000 7ff797fff640 1 -- 172.21.15.69:0/949814664 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+7821 (secure 0 0 0) 0x7ff7a8057a60 con 0x7ff7b00b4350 2023-10-30T20:23:53.312 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.310+0000 7ff797fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:23:53.312 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.310+0000 7ff797fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:23:53.313 INFO:teuthology.misc.health.smithi069.stdout: 2023-10-30T20:23:53.313 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":14,"fsid":"32c1a723-7694-47d8-8b37-b1257dfe1796","created":"2023-10-30T20:23:06.559937+0000","modified":"2023-10-30T20:23:52.372441+0000","last_up_change":"2023-10-30T20:23:52.372441+0000","last_in_change":"2023-10-30T20:23:12.023084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af10bc15-318a-4349-af76-32a30c8acf80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d0d32a9a-e24a-4d0c-9b90-125485d70672","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"508f2c3a-9e54-46aa-b6c0-52fad97fd341","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6805","nonce":2300065874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2300065874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2300065874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2300065874}]},"public_addr":"172.21.15.69:6805/2300065874","cluster_addr":"172.21.15.69:6806/2300065874","heartbeat_back_addr":"172.21.15.69:6808/2300065874","heartbeat_front_addr":"172.21.15.69:6807/2300065874","state":["exists","up"]},{"osd":3,"uuid":"003a2613-1050-4a96-b927-0e5983687319","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"678fc58e-ed6a-4fd0-9825-93297e3af6f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":4151811482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6801","nonce":4151811482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6803","nonce":4151811482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":4151811482}]},"public_addr":"172.21.15.183:6800/4151811482","cluster_addr":"172.21.15.183:6801/4151811482","heartbeat_back_addr":"172.21.15.183:6803/4151811482","heartbeat_front_addr":"172.21.15.183:6802/4151811482","state":["exists","up"]},{"osd":5,"uuid":"f945159a-a53c-48ba-97df-9374f9e56ffd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8184f59d-15e2-4bdc-90e3-fe1e231ca125","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1361328034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1361328034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":1361328034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1361328034}]},"public_addr":"172.21.15.183:6808/1361328034","cluster_addr":"172.21.15.183:6809/1361328034","heartbeat_back_addr":"172.21.15.183:6811/1361328034","heartbeat_front_addr":"172.21.15.183:6810/1361328034","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:48.903386+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T20:23:53.315 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 10 librados: watch_flush enter 2023-10-30T20:23:53.315 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 10 librados: watch_flush exit 2023-10-30T20:23:53.316 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 20 client.4217.objecter shutdown clearing up homeless session... 2023-10-30T20:23:53.316 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 10 client.4217.objecter successfully canceled tick 2023-10-30T20:23:53.316 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 1 -- 172.21.15.69:0/949814664 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff79c06c780 msgr2=0x7ff79c06ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:53.316 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 1 --2- 172.21.15.69:0/949814664 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff79c06c780 0x7ff79c06ec40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0036000 tx=0x7ff7a0008030 comp rx=0 tx=0).stop 2023-10-30T20:23:53.316 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 10 monclient: shutdown 2023-10-30T20:23:53.316 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:23:53.317 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 1 -- 172.21.15.69:0/949814664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b00b4350 msgr2=0x7ff7b01da2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:23:53.317 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 1 --2- 172.21.15.69:0/949814664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b00b4350 0x7ff7b01da2d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff7a800c780 tx=0x7ff7a800cc40 comp rx=0 tx=0).stop 2023-10-30T20:23:53.317 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 1 -- 172.21.15.69:0/949814664 shutdown_connections 2023-10-30T20:23:53.317 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 1 --2- 172.21.15.69:0/949814664 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff79c06c780 0x7ff79c06ec40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:53.317 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 1 --2- 172.21.15.69:0/949814664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b00b4350 0x7ff7b01da2d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:53.317 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 1 --2- 172.21.15.69:0/949814664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b00b0d50 0x7ff7b0140820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:53.318 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 1 --2- 172.21.15.69:0/949814664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b00adc70 0x7ff7b0135ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:23:53.318 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.313+0000 7ff7b624f640 1 -- 172.21.15.69:0/949814664 >> 172.21.15.69:0/949814664 conn(0x7ff7b0060bf0 msgr2=0x7ff7b013b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:23:53.318 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.314+0000 7ff7b624f640 1 -- 172.21.15.69:0/949814664 shutdown_connections 2023-10-30T20:23:53.318 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.314+0000 7ff7b624f640 1 -- 172.21.15.69:0/949814664 wait complete. 2023-10-30T20:23:53.318 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:23:53.314+0000 7ff7b624f640 1 librados: shutdown 2023-10-30T20:23:53.328 DEBUG:teuthology.misc:3 of 8 OSDs are up 2023-10-30T20:23:53.401 INFO:tasks.ceph.osd.3.smithi069.stderr:2023-10-30T20:23:53.382+0000 11a5d640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T20:23:53.402 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-10-30T20:23:53.388+0000 11a5d640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T20:23:53.680 INFO:tasks.ceph.osd.1.smithi069.stderr:2023-10-30T20:23:53.664+0000 1225e640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T20:24:00.133 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T20:24:00.223 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.222+0000 7f28d2905640 10 monclient: get_monmap_and_config 2023-10-30T20:24:00.223 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.222+0000 7f28d2905640 10 monclient: build_initial_monmap 2023-10-30T20:24:00.223 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.222+0000 7f28d2905640 10 monclient: monmap: 2023-10-30T20:24:00.224 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-10-30T20:24:00.224 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:00.224 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:24:00.223068+0000 2023-10-30T20:24:00.224 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:24:00.223068+0000 2023-10-30T20:24:00.224 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:00.225 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:00.225 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:00.225 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:00.225 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:00.225 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:00.226 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:00.226 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:24:00.227 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.222+0000 7f28d2905640 1 Processor -- start 2023-10-30T20:24:00.227 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.222+0000 7f28d2905640 1 -- start start 2023-10-30T20:24:00.227 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.222+0000 7f28d2905640 10 monclient: init 2023-10-30T20:24:00.228 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.223+0000 7f28d2905640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:00.228 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.223+0000 7f28d2905640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-10-30T20:24:00.228 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.223+0000 7f28d2905640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28cc0b4350 addr v1:172.21.15.69:6789/0 2023-10-30T20:24:00.228 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.223+0000 7f28d2905640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f28cc0adc70 0x7f28cc0b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:00.228 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.223+0000 7f28d2905640 10 monclient(hunting): picked mon.noname-a con 0x7f28cc0adc70 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:00.228 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.223+0000 7f28d2905640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28cc0b0d50 0x7f28cc13ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:00.229 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.223+0000 7f28d2905640 10 monclient(hunting): picked mon.noname-b con 0x7f28cc0b0d50 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:00.229 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.223+0000 7f28d2905640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28cc0ae950 con 0x7f28cc0b4350 2023-10-30T20:24:00.229 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.223+0000 7f28d2905640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:00.229 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.223+0000 7f28d2905640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28cc067a10 con 0x7f28cc0adc70 2023-10-30T20:24:00.229 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.223+0000 7f28d2905640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:00.230 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.223+0000 7f28d2905640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc0b4730 con 0x7f28cc0b0d50 2023-10-30T20:24:00.230 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.223+0000 7f28d2905640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:00.230 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.223+0000 7f28d2905640 10 monclient(hunting): authenticate will time out at 1563.512253s 2023-10-30T20:24:00.231 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.228+0000 7f28cbfff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f28cc0b4350 0x7f28cc0aee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47794/0 (socket says 172.21.15.69:47794) 2023-10-30T20:24:00.231 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.228+0000 7f28cbfff640 1 -- 172.21.15.69:0/1974629523 learned_addr learned my addr 172.21.15.69:0/1974629523 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:24:00.231 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.228+0000 7f28cb7fe640 1 --2- 172.21.15.69:0/1974629523 >> v2:172.21.15.69:3300/0 conn(0x7f28cc0adc70 0x7f28cc0b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:00.231 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.229+0000 7f28cb7fe640 10 monclient(hunting): get_auth_request con 0x7f28cc0adc70 auth_method 0 2023-10-30T20:24:00.231 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.229+0000 7f28cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:00.232 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.229+0000 7f28cb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:00.232 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.229+0000 7f28cb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:00.232 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.230+0000 7f28cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:00.232 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.230+0000 7f28cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:00.233 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.230+0000 7f28cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:00.233 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.231+0000 7f28caffd640 1 -- 172.21.15.69:0/1974629523 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932676651 0 0) 0x7f28bc001110 con 0x7f28cc0b4350 2023-10-30T20:24:00.233 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.231+0000 7f28caffd640 10 monclient(hunting): _negotiate 2023-10-30T20:24:00.234 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.231+0000 7f28caffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:00.234 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.231+0000 7f28caffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:00.234 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.231+0000 7f28caffd640 10 monclient(hunting): my global_id is 4245 2023-10-30T20:24:00.234 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.231+0000 7f28caffd640 1 -- 172.21.15.69:0/1974629523 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b4003520 con 0x7f28cc0b4350 2023-10-30T20:24:00.235 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.233+0000 7f28cb7fe640 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2023-10-30T20:24:00.235 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.233+0000 7f28cb7fe640 1 -- 172.21.15.69:0/1974629523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28cc0b0d50 msgr2=0x7f28cc13ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:00.235 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.233+0000 7f28cb7fe640 1 --2- 172.21.15.69:0/1974629523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28cc0b0d50 0x7f28cc13ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:00.235 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.233+0000 7f28cb7fe640 1 -- 172.21.15.69:0/1974629523 >> v1:172.21.15.69:6789/0 conn(0x7f28cc0b4350 legacy=0x7f28cc0aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:00.236 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.233+0000 7f28cb7fe640 10 monclient: _finish_hunting 0 2023-10-30T20:24:00.236 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.233+0000 7f28cb7fe640 1 monclient: found mon.noname-a 2023-10-30T20:24:00.236 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.233+0000 7f28cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:00.236 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.233+0000 7f28cb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:00.236 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.233+0000 7f28cb7fe640 1 -- 172.21.15.69:0/1974629523 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28cc0b3990 con 0x7f28cc0adc70 2023-10-30T20:24:00.237 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.236+0000 7f28cb7fe640 1 --2- 172.21.15.69:0/1974629523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28cc0adc70 0x7f28cc0b0810 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f28c00097a0 tx=0x7f28c002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d66906c9bb6399 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:00.239 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.238+0000 7f28caffd640 1 -- 172.21.15.69:0/1974629523 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c00315a0 con 0x7f28cc0adc70 2023-10-30T20:24:00.240 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.238+0000 7f28caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:00.240 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.238+0000 7f28caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:00.240 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.238+0000 7f28caffd640 10 monclient: dump: 2023-10-30T20:24:00.240 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:24:00.240 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:00.240 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:00.241 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:00.241 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:00.241 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:00.241 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:00.241 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:00.241 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:00.242 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:24:00.242 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.238+0000 7f28caffd640 10 monclient: _finish_auth 0 2023-10-30T20:24:00.242 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.238+0000 7f28caffd640 10 monclient: _check_auth_tickets 2023-10-30T20:24:00.242 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.238+0000 7f28caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:00.242 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.238+0000 7f28d2905640 5 monclient: authenticate success, global_id 4242 2023-10-30T20:24:00.243 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.238+0000 7f28d2905640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:00.245 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28caffd640 1 -- 172.21.15.69:0/1974629523 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c0031710 con 0x7f28cc0adc70 2023-10-30T20:24:00.246 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28caffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:00.246 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28caffd640 1 -- 172.21.15.69:0/1974629523 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0031a90 con 0x7f28cc0adc70 2023-10-30T20:24:00.246 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28d2905640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:00.246 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28d2905640 4 set_mon_vals no callback set 2023-10-30T20:24:00.246 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:00.247 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:00.247 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28caffd640 10 monclient: dump: 2023-10-30T20:24:00.247 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:24:00.247 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:00.248 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:00.248 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:00.248 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:00.248 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:00.248 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:00.248 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:00.249 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:00.249 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:24:00.249 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28d2905640 10 monclient: shutdown 2023-10-30T20:24:00.249 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28d2905640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:00.249 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28d2905640 1 -- 172.21.15.69:0/1974629523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28cc0adc70 msgr2=0x7f28cc0b0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:00.249 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28d2905640 1 --2- 172.21.15.69:0/1974629523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28cc0adc70 0x7f28cc0b0810 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f28c00097a0 tx=0x7f28c002b840 comp rx=0 tx=0).stop 2023-10-30T20:24:00.250 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28d2905640 1 -- 172.21.15.69:0/1974629523 shutdown_connections 2023-10-30T20:24:00.250 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28d2905640 1 --2- 172.21.15.69:0/1974629523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28cc0adc70 0x7f28cc0b0810 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:00.250 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28d2905640 1 --2- 172.21.15.69:0/1974629523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28cc0b0d50 0x7f28cc13ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:00.250 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28d2905640 1 -- 172.21.15.69:0/1974629523 >> 172.21.15.69:0/1974629523 conn(0x7f28cc060bf0 msgr2=0x7f28cc061000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:00.250 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28d2905640 1 -- 172.21.15.69:0/1974629523 shutdown_connections 2023-10-30T20:24:00.251 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.240+0000 7f28d2905640 1 -- 172.21.15.69:0/1974629523 wait complete. 2023-10-30T20:24:00.251 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.241+0000 7f28d2905640 10 monclient: build_initial_monmap 2023-10-30T20:24:00.251 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.241+0000 7f28d2905640 10 monclient: monmap: 2023-10-30T20:24:00.251 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-10-30T20:24:00.251 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:00.252 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:00.252 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2023-10-30T20:24:00.252 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:00.252 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:00.252 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:00.253 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:00.253 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:00.253 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:24:00.253 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.241+0000 7f28d2905640 1 librados: starting msgr at 2023-10-30T20:24:00.253 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.241+0000 7f28d2905640 1 librados: starting objecter 2023-10-30T20:24:00.254 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.241+0000 7f28d2905640 1 Processor -- start 2023-10-30T20:24:00.254 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 1 -- start start 2023-10-30T20:24:00.254 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 1 librados: setting wanted keys 2023-10-30T20:24:00.254 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 1 librados: calling monclient init 2023-10-30T20:24:00.254 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 10 monclient: init 2023-10-30T20:24:00.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:00.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:24:00.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28cc0adc70 0x7f28cc1de270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:00.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 10 monclient(hunting): picked mon.noname-a con 0x7f28cc0adc70 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:00.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28cc0b0d50 0x7f28cc1daec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:00.255 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 10 monclient(hunting): picked mon.noname-b con 0x7f28cc0b0d50 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:00.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28cc0b4350 0x7f28cc1db400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:00.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 10 monclient(hunting): picked mon.noname-c con 0x7f28cc0b4350 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:00.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:00.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc0b12e0 con 0x7f28cc0adc70 2023-10-30T20:24:00.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:00.256 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc06d630 con 0x7f28cc0b0d50 2023-10-30T20:24:00.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:00.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc06e290 con 0x7f28cc0b4350 2023-10-30T20:24:00.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:00.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.242+0000 7f28d2905640 10 monclient(hunting): authenticate will time out at 1563.531589s 2023-10-30T20:24:00.257 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.243+0000 7f28cbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28cc0adc70 0x7f28cc1de270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:00.258 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.243+0000 7f28cbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28cc0adc70 0x7f28cc1de270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45090/0 (socket says 172.21.15.69:45090) 2023-10-30T20:24:00.258 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.243+0000 7f28cbfff640 1 -- 172.21.15.69:0/3854464077 learned_addr learned my addr 172.21.15.69:0/3854464077 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:00.258 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.243+0000 7f28cbfff640 10 monclient(hunting): get_auth_request con 0x7f28cc0adc70 auth_method 0 2023-10-30T20:24:00.258 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.243+0000 7f28cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:00.258 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.243+0000 7f28cbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:00.258 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.243+0000 7f28cbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:00.259 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.243+0000 7f28cb7fe640 1 --2- 172.21.15.69:0/3854464077 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28cc0b0d50 0x7f28cc1daec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:00.259 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.243+0000 7f28cb7fe640 10 monclient(hunting): get_auth_request con 0x7f28cc0b0d50 auth_method 0 2023-10-30T20:24:00.259 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.243+0000 7f28cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:00.259 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.243+0000 7f28cb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:00.259 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.243+0000 7f28cb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:00.259 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.244+0000 7f28cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:00.260 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.244+0000 7f28cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:00.260 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.244+0000 7f28cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:00.260 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.244+0000 7f28cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:00.260 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.244+0000 7f28cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:00.260 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.244+0000 7f28cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:00.262 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.247+0000 7f28cb7fe640 10 monclient(hunting): handle_auth_done global_id 4234 payload 824 2023-10-30T20:24:00.262 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.247+0000 7f28cb7fe640 1 -- 172.21.15.69:0/3854464077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28cc0b4350 msgr2=0x7f28cc1db400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:00.262 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.247+0000 7f28cb7fe640 1 --2- 172.21.15.69:0/3854464077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28cc0b4350 0x7f28cc1db400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:00.263 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.247+0000 7f28cb7fe640 1 -- 172.21.15.69:0/3854464077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28cc0adc70 msgr2=0x7f28cc1de270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:00.263 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.247+0000 7f28cb7fe640 1 --2- 172.21.15.69:0/3854464077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28cc0adc70 0x7f28cc1de270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:00.263 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.247+0000 7f28cb7fe640 10 monclient: _finish_hunting 0 2023-10-30T20:24:00.263 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.247+0000 7f28cb7fe640 1 monclient: found mon.noname-b 2023-10-30T20:24:00.263 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.247+0000 7f28cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:00.264 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.247+0000 7f28cb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:24:00.264 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.247+0000 7f28cb7fe640 1 -- 172.21.15.69:0/3854464077 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28cc06e500 con 0x7f28cc0b0d50 2023-10-30T20:24:00.264 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.249+0000 7f28cb7fe640 1 --2- 172.21.15.69:0/3854464077 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28cc0b0d50 0x7f28cc1daec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f28c0033000 tx=0x7f28c00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:00.264 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.249+0000 7f28c8ff9640 10 client.?.objecter ms_handle_connect 0x7f28cc0b0d50 2023-10-30T20:24:00.264 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.249+0000 7f28c8ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:00.265 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28c8ff9640 1 -- 172.21.15.69:0/3854464077 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0040070 con 0x7f28cc0b0d50 2023-10-30T20:24:00.265 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:00.265 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28c8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:00.265 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28c8ff9640 10 monclient: dump: 2023-10-30T20:24:00.265 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:24:00.266 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:00.266 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:00.266 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:00.266 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:00.266 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:00.266 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:00.267 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:00.267 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:00.267 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:24:00.267 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28c8ff9640 10 monclient: _finish_auth 0 2023-10-30T20:24:00.267 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28c8ff9640 10 monclient: _check_auth_tickets 2023-10-30T20:24:00.268 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:00.268 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28c8ff9640 1 -- 172.21.15.69:0/3854464077 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c0003a20 con 0x7f28cc0b0d50 2023-10-30T20:24:00.268 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28d2905640 5 monclient: authenticate success, global_id 4234 2023-10-30T20:24:00.268 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28d2905640 10 monclient: _renew_subs 2023-10-30T20:24:00.268 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28d2905640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:00.268 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28d2905640 1 -- 172.21.15.69:0/3854464077 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28cc1db940 con 0x7f28cc0b0d50 2023-10-30T20:24:00.269 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28c8ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:00.269 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28c8ff9640 1 -- 172.21.15.69:0/3854464077 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c00315a0 con 0x7f28cc0b0d50 2023-10-30T20:24:00.269 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:00.269 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28c8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:00.269 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28c8ff9640 10 monclient: dump: 2023-10-30T20:24:00.269 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-10-30T20:24:00.270 INFO:teuthology.misc.health.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:00.270 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:00.270 INFO:teuthology.misc.health.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:00.270 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:00.270 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:00.271 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:00.271 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:00.271 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:00.271 INFO:teuthology.misc.health.smithi069.stderr: 2023-10-30T20:24:00.271 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28ca7fc640 4 set_mon_vals no callback set 2023-10-30T20:24:00.272 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28d2905640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:00.272 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28d2905640 10 monclient: _renew_subs 2023-10-30T20:24:00.272 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28d2905640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:00.272 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28d2905640 1 -- 172.21.15.69:0/3854464077 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28cc13add0 con 0x7f28cc0b0d50 2023-10-30T20:24:00.272 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.253+0000 7f28d2905640 1 librados: init done 2023-10-30T20:24:00.272 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.254+0000 7f28d2905640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:00.273 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.254+0000 7f28d2905640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:00.273 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.254+0000 7f28d2905640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:00.273 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.254+0000 7f28d2905640 1 -- 172.21.15.69:0/3854464077 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28cc04e9b0 con 0x7f28cc0b0d50 2023-10-30T20:24:00.498 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.497+0000 7f28c8ff9640 1 -- 172.21.15.69:0/3854464077 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7f28c0047480 con 0x7f28cc0b0d50 2023-10-30T20:24:00.499 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.497+0000 7f28c8ff9640 1 --2- 172.21.15.69:0/3854464077 >> v2:172.21.15.69:6800/1876307353 conn(0x7f28b806c7b0 0x7f28b806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:00.499 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.497+0000 7f28cbfff640 1 --2- 172.21.15.69:0/3854464077 >> v2:172.21.15.69:6800/1876307353 conn(0x7f28b806c7b0 0x7f28b806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:00.499 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.498+0000 7f28cbfff640 10 monclient: get_auth_request con 0x7f28b806c7b0 auth_method 0 2023-10-30T20:24:00.500 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.498+0000 7f28cbfff640 1 --2- 172.21.15.69:0/3854464077 >> v2:172.21.15.69:6800/1876307353 conn(0x7f28b806c7b0 0x7f28b806ec70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f28bc001380 tx=0x7f28bc007c50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:00.500 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.498+0000 7f28c8ff9640 10 client.4234.objecter ms_handle_connect 0x7f28b806c7b0 2023-10-30T20:24:00.501 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.499+0000 7f28c8ff9640 1 -- 172.21.15.69:0/3854464077 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3737+0+0 (secure 0 0 0) 0x7f28c007b100 con 0x7f28cc0b0d50 2023-10-30T20:24:00.501 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.499+0000 7f28c8ff9640 10 client.4234.objecter ms_dispatch 0x7f28cc000b90 osd_map(18..18 src has 1..18) v4 2023-10-30T20:24:00.501 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.499+0000 7f28c8ff9640 3 client.4234.objecter handle_osd_map got epochs [18,18] > 0 2023-10-30T20:24:00.501 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.499+0000 7f28c8ff9640 3 client.4234.objecter handle_osd_map decoding full epoch 18 2023-10-30T20:24:00.501 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.500+0000 7f28c8ff9640 20 client.4234.objecter dump_active .. 0 homeless 2023-10-30T20:24:00.510 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.509+0000 7f28c8ff9640 1 -- 172.21.15.69:0/3854464077 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f28c0037070 con 0x7f28cc0b0d50 2023-10-30T20:24:00.510 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.509+0000 7f28c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:00.511 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.509+0000 7f28c8ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:00.711 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.710+0000 7f28d2905640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:24:00.712 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.710+0000 7f28d2905640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:24:00.712 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.710+0000 7f28d2905640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:00.712 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:00.710+0000 7f28d2905640 1 -- 172.21.15.69:0/3854464077 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28cc06e500 con 0x7f28cc0b0d50 2023-10-30T20:24:01.020 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.019+0000 7f28c8ff9640 1 -- 172.21.15.69:0/3854464077 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f28c00767f0 con 0x7f28cc0b0d50 2023-10-30T20:24:01.024 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.022+0000 7f28c8ff9640 1 -- 172.21.15.69:0/3854464077 <== mon.1 v2:172.21.15.183:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+10918 (secure 0 0 0) 0x7f28c00093c0 con 0x7f28cc0b0d50 2023-10-30T20:24:01.024 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.022+0000 7f28c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:24:01.024 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.022+0000 7f28c8ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:01.024 INFO:teuthology.misc.health.smithi069.stdout: 2023-10-30T20:24:01.025 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":18,"fsid":"32c1a723-7694-47d8-8b37-b1257dfe1796","created":"2023-10-30T20:23:06.559937+0000","modified":"2023-10-30T20:23:58.253207+0000","last_up_change":"2023-10-30T20:23:54.534733+0000","last_in_change":"2023-10-30T20:23:12.023084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T20:23:54.411443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"af10bc15-318a-4349-af76-32a30c8acf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6801","nonce":2068724062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2068724062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":2068724062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2068724062}]},"public_addr":"172.21.15.69:6801/2068724062","cluster_addr":"172.21.15.69:6802/2068724062","heartbeat_back_addr":"172.21.15.69:6804/2068724062","heartbeat_front_addr":"172.21.15.69:6803/2068724062","state":["exists","up"]},{"osd":1,"uuid":"d0d32a9a-e24a-4d0c-9b90-125485d70672","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6813","nonce":431025540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":431025540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":431025540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":431025540}]},"public_addr":"172.21.15.69:6813/431025540","cluster_addr":"172.21.15.69:6814/431025540","heartbeat_back_addr":"172.21.15.69:6816/431025540","heartbeat_front_addr":"172.21.15.69:6815/431025540","state":["exists","up"]},{"osd":2,"uuid":"508f2c3a-9e54-46aa-b6c0-52fad97fd341","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6805","nonce":2300065874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2300065874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2300065874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2300065874}]},"public_addr":"172.21.15.69:6805/2300065874","cluster_addr":"172.21.15.69:6806/2300065874","heartbeat_back_addr":"172.21.15.69:6808/2300065874","heartbeat_front_addr":"172.21.15.69:6807/2300065874","state":["exists","up"]},{"osd":3,"uuid":"003a2613-1050-4a96-b927-0e5983687319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6809","nonce":1824616966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1824616966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1824616966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1824616966}]},"public_addr":"172.21.15.69:6809/1824616966","cluster_addr":"172.21.15.69:6810/1824616966","heartbeat_back_addr":"172.21.15.69:6812/1824616966","heartbeat_front_addr":"172.21.15.69:6811/1824616966","state":["exists","up"]},{"osd":4,"uuid":"678fc58e-ed6a-4fd0-9825-93297e3af6f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":4151811482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6801","nonce":4151811482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6803","nonce":4151811482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":4151811482}]},"public_addr":"172.21.15.183:6800/4151811482","cluster_addr":"172.21.15.183:6801/4151811482","heartbeat_back_addr":"172.21.15.183:6803/4151811482","heartbeat_front_addr":"172.21.15.183:6802/4151811482","state":["exists","up"]},{"osd":5,"uuid":"f945159a-a53c-48ba-97df-9374f9e56ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2276632263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6813","nonce":2276632263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6815","nonce":2276632263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2276632263}]},"public_addr":"172.21.15.183:6812/2276632263","cluster_addr":"172.21.15.183:6813/2276632263","heartbeat_back_addr":"172.21.15.183:6815/2276632263","heartbeat_front_addr":"172.21.15.183:6814/2276632263","state":["exists","up"]},{"osd":6,"uuid":"8184f59d-15e2-4bdc-90e3-fe1e231ca125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1046221745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":1046221745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6807","nonce":1046221745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1046221745}]},"public_addr":"172.21.15.183:6804/1046221745","cluster_addr":"172.21.15.183:6805/1046221745","heartbeat_back_addr":"172.21.15.183:6807/1046221745","heartbeat_front_addr":"172.21.15.183:6806/1046221745","state":["exists","up"]},{"osd":7,"uuid":"1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1361328034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1361328034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":1361328034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1361328034}]},"public_addr":"172.21.15.183:6808/1361328034","cluster_addr":"172.21.15.183:6809/1361328034","heartbeat_back_addr":"172.21.15.183:6811/1361328034","heartbeat_front_addr":"172.21.15.183:6810/1361328034","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.509883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.602550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:50.691692+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.518736+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:48.903386+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.213129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:50.677675+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:50.250250+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T20:24:01.026 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 10 librados: watch_flush enter 2023-10-30T20:24:01.026 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 10 librados: watch_flush exit 2023-10-30T20:24:01.027 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 20 client.4234.objecter shutdown clearing up homeless session... 2023-10-30T20:24:01.027 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 10 client.4234.objecter successfully canceled tick 2023-10-30T20:24:01.027 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 1 -- 172.21.15.69:0/3854464077 >> v2:172.21.15.69:6800/1876307353 conn(0x7f28b806c7b0 msgr2=0x7f28b806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:01.027 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 1 --2- 172.21.15.69:0/3854464077 >> v2:172.21.15.69:6800/1876307353 conn(0x7f28b806c7b0 0x7f28b806ec70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f28bc001380 tx=0x7f28bc007c50 comp rx=0 tx=0).stop 2023-10-30T20:24:01.027 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 10 monclient: shutdown 2023-10-30T20:24:01.028 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:01.028 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 1 -- 172.21.15.69:0/3854464077 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28cc0b0d50 msgr2=0x7f28cc1daec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:01.028 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 1 --2- 172.21.15.69:0/3854464077 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28cc0b0d50 0x7f28cc1daec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f28c0033000 tx=0x7f28c00046c0 comp rx=0 tx=0).stop 2023-10-30T20:24:01.028 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 1 -- 172.21.15.69:0/3854464077 shutdown_connections 2023-10-30T20:24:01.028 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 1 --2- 172.21.15.69:0/3854464077 >> v2:172.21.15.69:6800/1876307353 conn(0x7f28b806c7b0 0x7f28b806ec70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:01.029 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 1 --2- 172.21.15.69:0/3854464077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28cc0b4350 0x7f28cc1db400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:01.029 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 1 --2- 172.21.15.69:0/3854464077 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28cc0b0d50 0x7f28cc1daec0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:01.029 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 1 --2- 172.21.15.69:0/3854464077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28cc0adc70 0x7f28cc1de270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:01.029 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 1 -- 172.21.15.69:0/3854464077 >> 172.21.15.69:0/3854464077 conn(0x7f28cc060bf0 msgr2=0x7f28cc0737b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:01.029 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.024+0000 7f28d2905640 1 -- 172.21.15.69:0/3854464077 shutdown_connections 2023-10-30T20:24:01.029 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.025+0000 7f28d2905640 1 -- 172.21.15.69:0/3854464077 wait complete. 2023-10-30T20:24:01.030 INFO:teuthology.misc.health.smithi069.stderr:2023-10-30T20:24:01.025+0000 7f28d2905640 1 librados: shutdown 2023-10-30T20:24:01.036 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-30T20:24:01.036 INFO:tasks.ceph:Creating RBD pool 2023-10-30T20:24:01.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-30T20:24:01.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.133+0000 7f78a5747640 10 monclient: get_monmap_and_config 2023-10-30T20:24:01.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.133+0000 7f78a5747640 10 monclient: build_initial_monmap 2023-10-30T20:24:01.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.133+0000 7f78a5747640 10 monclient: monmap: 2023-10-30T20:24:01.136 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:01.136 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:01.136 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:01.134897+0000 2023-10-30T20:24:01.136 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:01.134897+0000 2023-10-30T20:24:01.136 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:01.136 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:01.138 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:01.139 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.134+0000 7f78a5747640 1 Processor -- start 2023-10-30T20:24:01.140 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.134+0000 7f78a5747640 1 -- start start 2023-10-30T20:24:01.140 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.134+0000 7f78a5747640 10 monclient: init 2023-10-30T20:24:01.140 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f78a5747640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:01.140 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f78a5747640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-10-30T20:24:01.140 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f78a5747640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f78a00b4440 0x7f78a0144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:01.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f78a5747640 10 monclient(hunting): picked mon.noname-c con 0x7f78a00b4440 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:01.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f78a5747640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78a01453e0 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:01.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f78a5747640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78a00b2230 0x7f78a014c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:01.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f78a5747640 10 monclient(hunting): picked mon.noname-a con 0x7f78a00b2230 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:01.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f78a5747640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a00a9da0 con 0x7f78a01453e0 2023-10-30T20:24:01.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f78a5747640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:01.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f78a5747640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a0067a10 con 0x7f78a00b2230 2023-10-30T20:24:01.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f78a5747640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:01.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f78a5747640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a00aa080 con 0x7f78a00b4440 2023-10-30T20:24:01.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f78a5747640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:01.143 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f78a5747640 10 monclient(hunting): authenticate will time out at 1564.424028s 2023-10-30T20:24:01.143 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.135+0000 7f789f7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78a00b2230 0x7f78a014c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:01.143 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.136+0000 7f789f7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78a00b2230 0x7f78a014c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45106/0 (socket says 172.21.15.69:45106) 2023-10-30T20:24:01.143 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.136+0000 7f789f7fe640 1 -- 172.21.15.69:0/1174737422 learned_addr learned my addr 172.21.15.69:0/1174737422 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:01.143 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.136+0000 7f789f7fe640 10 monclient(hunting): get_auth_request con 0x7f78a00b2230 auth_method 0 2023-10-30T20:24:01.143 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.136+0000 7f789f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:01.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.136+0000 7f789f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:01.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.136+0000 7f789f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:01.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.136+0000 7f789f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:01.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.136+0000 7f789f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:01.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.136+0000 7f789f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:01.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.137+0000 7f789effd640 1 --2- 172.21.15.69:0/1174737422 >> v2:172.21.15.183:3300/0 conn(0x7f78a00b4440 0x7f78a0144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:01.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.138+0000 7f789f7fe640 10 monclient(hunting): handle_auth_done global_id 4251 payload 274 2023-10-30T20:24:01.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.138+0000 7f789f7fe640 1 -- 172.21.15.69:0/1174737422 >> v2:172.21.15.183:3300/0 conn(0x7f78a00b4440 msgr2=0x7f78a0144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:01.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.138+0000 7f789f7fe640 1 --2- 172.21.15.69:0/1174737422 >> v2:172.21.15.183:3300/0 conn(0x7f78a00b4440 0x7f78a0144ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:01.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.138+0000 7f789f7fe640 1 -- 172.21.15.69:0/1174737422 >> v1:172.21.15.183:6789/0 conn(0x7f78a01453e0 legacy=0x7f78a00b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:01.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.138+0000 7f789f7fe640 10 monclient: _finish_hunting 0 2023-10-30T20:24:01.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.138+0000 7f789f7fe640 1 monclient: found mon.noname-a 2023-10-30T20:24:01.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.138+0000 7f789f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:01.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.138+0000 7f789f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:01.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.138+0000 7f789f7fe640 1 -- 172.21.15.69:0/1174737422 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00aa360 con 0x7f78a00b2230 2023-10-30T20:24:01.150 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.140+0000 7f789f7fe640 1 --2- 172.21.15.69:0/1174737422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78a00b2230 0x7f78a014c3e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f78a0066a50 tx=0x7f789802bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2acba1b892da982 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:01.150 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f789dffb640 1 -- 172.21.15.69:0/1174737422 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789802c960 con 0x7f78a00b2230 2023-10-30T20:24:01.150 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f789dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:01.150 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f789dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:01.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f789dffb640 10 monclient: dump: 2023-10-30T20:24:01.151 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:01.151 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:01.151 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:01.151 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:01.151 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:01.152 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:01.152 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:01.152 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:01.152 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:01.152 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:01.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f789dffb640 10 monclient: _finish_auth 0 2023-10-30T20:24:01.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f789dffb640 10 monclient: _check_auth_tickets 2023-10-30T20:24:01.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f789dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:01.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f789dffb640 1 -- 172.21.15.69:0/1174737422 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789802cad0 con 0x7f78a00b2230 2023-10-30T20:24:01.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f78a5747640 5 monclient: authenticate success, global_id 4251 2023-10-30T20:24:01.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f78a5747640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:01.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f789dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:01.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f78a5747640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:01.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f78a5747640 4 set_mon_vals no callback set 2023-10-30T20:24:01.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f789dffb640 1 -- 172.21.15.69:0/1174737422 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789802ce70 con 0x7f78a00b2230 2023-10-30T20:24:01.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f789dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:01.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f789dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:01.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.143+0000 7f789dffb640 10 monclient: dump: 2023-10-30T20:24:01.155 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:01.155 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:01.155 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:01.156 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:01.156 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:01.156 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:01.156 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:01.156 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:01.156 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:01.157 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:01.157 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.144+0000 7f78a5747640 10 monclient: shutdown 2023-10-30T20:24:01.157 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.144+0000 7f78a5747640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:01.157 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.144+0000 7f78a5747640 1 -- 172.21.15.69:0/1174737422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78a00b2230 msgr2=0x7f78a014c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:01.157 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.144+0000 7f78a5747640 1 --2- 172.21.15.69:0/1174737422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78a00b2230 0x7f78a014c3e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f78a0066a50 tx=0x7f789802bdc0 comp rx=0 tx=0).stop 2023-10-30T20:24:01.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.144+0000 7f78a5747640 1 -- 172.21.15.69:0/1174737422 shutdown_connections 2023-10-30T20:24:01.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.144+0000 7f78a5747640 1 --2- 172.21.15.69:0/1174737422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78a00b2230 0x7f78a014c3e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:01.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.144+0000 7f78a5747640 1 --2- 172.21.15.69:0/1174737422 >> v2:172.21.15.183:3300/0 conn(0x7f78a00b4440 0x7f78a0144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:01.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.144+0000 7f78a5747640 1 -- 172.21.15.69:0/1174737422 >> 172.21.15.69:0/1174737422 conn(0x7f78a0065ea0 msgr2=0x7f78a00662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:01.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.144+0000 7f78a5747640 1 -- 172.21.15.69:0/1174737422 shutdown_connections 2023-10-30T20:24:01.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.144+0000 7f78a5747640 1 -- 172.21.15.69:0/1174737422 wait complete. 2023-10-30T20:24:01.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.145+0000 7f78a5747640 10 monclient: build_initial_monmap 2023-10-30T20:24:01.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.145+0000 7f78a5747640 10 monclient: monmap: 2023-10-30T20:24:01.159 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:01.159 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:01.159 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:01.160 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:01.160 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:01.160 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:01.160 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:01.160 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:01.161 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:01.161 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:01.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.145+0000 7f78a5747640 1 librados: starting msgr at 2023-10-30T20:24:01.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.145+0000 7f78a5747640 1 librados: starting objecter 2023-10-30T20:24:01.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.145+0000 7f78a5747640 1 Processor -- start 2023-10-30T20:24:01.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.145+0000 7f78a5747640 1 -- start start 2023-10-30T20:24:01.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.145+0000 7f78a5747640 1 librados: setting wanted keys 2023-10-30T20:24:01.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.145+0000 7f78a5747640 1 librados: calling monclient init 2023-10-30T20:24:01.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.145+0000 7f78a5747640 10 monclient: init 2023-10-30T20:24:01.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:01.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T20:24:01.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78a00b2230 0x7f78a00ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:01.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 10 monclient(hunting): picked mon.noname-c con 0x7f78a00b2230 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:01.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f78a00b4440 0x7f78a01d6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:01.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 10 monclient(hunting): picked mon.noname-b con 0x7f78a00b4440 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:01.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78a01453e0 0x7f78a01d6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:01.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 10 monclient(hunting): picked mon.noname-a con 0x7f78a01453e0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:01.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:01.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a013c220 con 0x7f78a01453e0 2023-10-30T20:24:01.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:01.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00b0510 con 0x7f78a00b4440 2023-10-30T20:24:01.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:01.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00ad550 con 0x7f78a00b2230 2023-10-30T20:24:01.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:01.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.146+0000 7f78a5747640 10 monclient(hunting): authenticate will time out at 1564.435301s 2023-10-30T20:24:01.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78a01453e0 0x7f78a01d6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:01.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789effd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78a00b2230 0x7f78a00ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789e7fc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f78a00b4440 0x7f78a01d6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789e7fc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f78a00b4440 0x7f78a01d6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36334/0 (socket says 172.21.15.69:36334) 2023-10-30T20:24:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789e7fc640 1 -- 172.21.15.69:0/4199695179 learned_addr learned my addr 172.21.15.69:0/4199695179 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789e7fc640 10 monclient(hunting): get_auth_request con 0x7f78a00b4440 auth_method 0 2023-10-30T20:24:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:01.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:01.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:01.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789f7fe640 10 monclient(hunting): get_auth_request con 0x7f78a01453e0 auth_method 0 2023-10-30T20:24:01.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:01.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:01.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:01.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789effd640 10 monclient(hunting): get_auth_request con 0x7f78a00b2230 auth_method 0 2023-10-30T20:24:01.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:01.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:01.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:01.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.147+0000 7f789e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.148+0000 7f789f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.148+0000 7f789f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.148+0000 7f789f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.148+0000 7f789effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.148+0000 7f789effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.148+0000 7f789effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:01.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.150+0000 7f789e7fc640 10 monclient(hunting): handle_auth_done global_id 4237 payload 824 2023-10-30T20:24:01.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.150+0000 7f789e7fc640 1 -- 172.21.15.69:0/4199695179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78a00b2230 msgr2=0x7f78a00ac8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:01.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.150+0000 7f789e7fc640 1 --2- 172.21.15.69:0/4199695179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78a00b2230 0x7f78a00ac8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:01.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.150+0000 7f789e7fc640 1 -- 172.21.15.69:0/4199695179 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78a01453e0 msgr2=0x7f78a01d6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:01.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.150+0000 7f789e7fc640 1 --2- 172.21.15.69:0/4199695179 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78a01453e0 0x7f78a01d6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:01.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.150+0000 7f789e7fc640 10 monclient: _finish_hunting 0 2023-10-30T20:24:01.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.150+0000 7f789e7fc640 1 monclient: found mon.noname-b 2023-10-30T20:24:01.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.150+0000 7f789e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:01.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.150+0000 7f789e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:24:01.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.150+0000 7f789e7fc640 1 -- 172.21.15.69:0/4199695179 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00b0170 con 0x7f78a00b4440 2023-10-30T20:24:01.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.152+0000 7f789e7fc640 1 --2- 172.21.15.69:0/4199695179 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f78a00b4440 0x7f78a01d6560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f789000b3e0 tx=0x7f789000b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:01.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.152+0000 7f78877fe640 10 client.?.objecter ms_handle_connect 0x7f78a00b4440 2023-10-30T20:24:01.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.152+0000 7f78877fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:01.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78877fe640 1 -- 172.21.15.69:0/4199695179 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890011830 con 0x7f78a00b4440 2023-10-30T20:24:01.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:01.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:01.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78877fe640 10 monclient: dump: 2023-10-30T20:24:01.175 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:01.175 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:01.175 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:01.175 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:01.175 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:01.176 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:01.176 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:01.176 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:01.176 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:01.176 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:01.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78877fe640 10 monclient: _finish_auth 0 2023-10-30T20:24:01.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78877fe640 10 monclient: _check_auth_tickets 2023-10-30T20:24:01.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:01.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78877fe640 1 -- 172.21.15.69:0/4199695179 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789000d040 con 0x7f78a00b4440 2023-10-30T20:24:01.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78877fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:01.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78877fe640 1 -- 172.21.15.69:0/4199695179 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890011c00 con 0x7f78a00b4440 2023-10-30T20:24:01.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78a5747640 5 monclient: authenticate success, global_id 4237 2023-10-30T20:24:01.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78a5747640 10 monclient: _renew_subs 2023-10-30T20:24:01.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78a5747640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:01.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78a5747640 1 -- 172.21.15.69:0/4199695179 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a00b0a30 con 0x7f78a00b4440 2023-10-30T20:24:01.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:01.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:01.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78877fe640 10 monclient: dump: 2023-10-30T20:24:01.179 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:01.179 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:01.179 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:01.179 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:01.180 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:01.180 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:01.180 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:01.180 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:01.180 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:01.181 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:01.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f789d7fa640 4 set_mon_vals no callback set 2023-10-30T20:24:01.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78a5747640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:01.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78a5747640 10 monclient: _renew_subs 2023-10-30T20:24:01.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78a5747640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:01.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78a5747640 1 -- 172.21.15.69:0/4199695179 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a00af920 con 0x7f78a00b4440 2023-10-30T20:24:01.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.156+0000 7f78a5747640 1 librados: init done 2023-10-30T20:24:01.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.157+0000 7f78a5747640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:01.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.157+0000 7f78a5747640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:01.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.157+0000 7f78a5747640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:01.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.157+0000 7f78a5747640 1 -- 172.21.15.69:0/4199695179 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a00b0370 con 0x7f78a00b4440 2023-10-30T20:24:01.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.175+0000 7f78877fe640 1 -- 172.21.15.69:0/4199695179 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f789001a400 con 0x7f78a00b4440 2023-10-30T20:24:01.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.176+0000 7f78877fe640 1 --2- 172.21.15.69:0/4199695179 >> v2:172.21.15.69:6800/1876307353 conn(0x7f788c06c7c0 0x7f788c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:01.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.176+0000 7f789effd640 1 --2- 172.21.15.69:0/4199695179 >> v2:172.21.15.69:6800/1876307353 conn(0x7f788c06c7c0 0x7f788c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:01.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.176+0000 7f789effd640 10 monclient: get_auth_request con 0x7f788c06c7c0 auth_method 0 2023-10-30T20:24:01.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.176+0000 7f78877fe640 10 client.4237.objecter ms_handle_connect 0x7f788c06c7c0 2023-10-30T20:24:01.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.176+0000 7f789effd640 1 --2- 172.21.15.69:0/4199695179 >> v2:172.21.15.69:6800/1876307353 conn(0x7f788c06c7c0 0x7f788c06ec80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f789400aac0 tx=0x7f7894008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:01.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.178+0000 7f78877fe640 1 -- 172.21.15.69:0/4199695179 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3737+0+0 (secure 0 0 0) 0x7f7890058250 con 0x7f78a00b4440 2023-10-30T20:24:01.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.178+0000 7f78877fe640 10 client.4237.objecter ms_dispatch 0x7f78a0000b90 osd_map(18..18 src has 1..18) v4 2023-10-30T20:24:01.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.178+0000 7f78877fe640 3 client.4237.objecter handle_osd_map got epochs [18,18] > 0 2023-10-30T20:24:01.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.178+0000 7f78877fe640 3 client.4237.objecter handle_osd_map decoding full epoch 18 2023-10-30T20:24:01.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.178+0000 7f78877fe640 20 client.4237.objecter dump_active .. 0 homeless 2023-10-30T20:24:01.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.411+0000 7f78877fe640 1 -- 172.21.15.69:0/4199695179 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f789005c2f0 con 0x7f78a00b4440 2023-10-30T20:24:01.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.411+0000 7f78877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:01.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.411+0000 7f78877fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:01.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.600+0000 7f78a5747640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T20:24:01.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.600+0000 7f78a5747640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T20:24:01.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.600+0000 7f78a5747640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:01.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:01.600+0000 7f78a5747640 1 -- 172.21.15.69:0/4199695179 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f78a00b0a30 con 0x7f78a00b4440 2023-10-30T20:24:02.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.702+0000 7f78877fe640 1 -- 172.21.15.69:0/4199695179 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v19) v1 ==== 109+0+0 (secure 0 0 0) 0x7f78900604a0 con 0x7f78a00b4440 2023-10-30T20:24:02.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.702+0000 7f78877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T20:24:02.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.702+0000 7f78877fe640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-10-30T20:24:02.705 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'rbd' created 2023-10-30T20:24:02.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 10 librados: watch_flush enter 2023-10-30T20:24:02.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 10 librados: watch_flush exit 2023-10-30T20:24:02.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 20 client.4237.objecter shutdown clearing up homeless session... 2023-10-30T20:24:02.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 10 client.4237.objecter successfully canceled tick 2023-10-30T20:24:02.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 1 -- 172.21.15.69:0/4199695179 >> v2:172.21.15.69:6800/1876307353 conn(0x7f788c06c7c0 msgr2=0x7f788c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:02.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 1 --2- 172.21.15.69:0/4199695179 >> v2:172.21.15.69:6800/1876307353 conn(0x7f788c06c7c0 0x7f788c06ec80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f789400aac0 tx=0x7f7894008040 comp rx=0 tx=0).stop 2023-10-30T20:24:02.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 10 monclient: shutdown 2023-10-30T20:24:02.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:02.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 1 -- 172.21.15.69:0/4199695179 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f78a00b4440 msgr2=0x7f78a01d6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:02.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 1 --2- 172.21.15.69:0/4199695179 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f78a00b4440 0x7f78a01d6560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f789000b3e0 tx=0x7f789000b8a0 comp rx=0 tx=0).stop 2023-10-30T20:24:02.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 1 -- 172.21.15.69:0/4199695179 shutdown_connections 2023-10-30T20:24:02.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 1 --2- 172.21.15.69:0/4199695179 >> v2:172.21.15.69:6800/1876307353 conn(0x7f788c06c7c0 0x7f788c06ec80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:02.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 1 --2- 172.21.15.69:0/4199695179 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78a01453e0 0x7f78a01d6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:02.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 1 --2- 172.21.15.69:0/4199695179 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f78a00b4440 0x7f78a01d6560 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:02.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 1 --2- 172.21.15.69:0/4199695179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78a00b2230 0x7f78a00ac8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:02.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 1 -- 172.21.15.69:0/4199695179 >> 172.21.15.69:0/4199695179 conn(0x7f78a0065ea0 msgr2=0x7f78a00b78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:02.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 1 -- 172.21.15.69:0/4199695179 shutdown_connections 2023-10-30T20:24:02.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 1 -- 172.21.15.69:0/4199695179 wait complete. 2023-10-30T20:24:02.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.705+0000 7f78a5747640 1 librados: shutdown 2023-10-30T20:24:02.719 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-30T20:24:02.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.819+0000 7f3a7c2fc640 10 monclient: get_monmap_and_config 2023-10-30T20:24:02.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.819+0000 7f3a7c2fc640 10 monclient: build_initial_monmap 2023-10-30T20:24:02.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.819+0000 7f3a7c2fc640 10 monclient: monmap: 2023-10-30T20:24:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:02.820098+0000 2023-10-30T20:24:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:02.820098+0000 2023-10-30T20:24:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:02.822 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:02.822 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:02.822 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:02.822 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:02.822 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:02.822 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:02.824 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.819+0000 7f3a7c2fc640 1 Processor -- start 2023-10-30T20:24:02.824 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.819+0000 7f3a7c2fc640 1 -- start start 2023-10-30T20:24:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.819+0000 7f3a7c2fc640 10 monclient: init 2023-10-30T20:24:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.820+0000 7f3a7c2fc640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.820+0000 7f3a7c2fc640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-10-30T20:24:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.820+0000 7f3a7c2fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a740b4420 addr v1:172.21.15.69:6789/0 2023-10-30T20:24:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.820+0000 7f3a7c2fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a740b12d0 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.820+0000 7f3a7c2fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7413f160 0x7f3a74142020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:02.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.820+0000 7f3a7c2fc640 10 monclient(hunting): picked mon.noname-b con 0x7f3a7413f160 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:02.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.820+0000 7f3a7c2fc640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7413f580 con 0x7f3a740b4420 2023-10-30T20:24:02.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.820+0000 7f3a7c2fc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a740a9120 con 0x7f3a740b12d0 2023-10-30T20:24:02.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.820+0000 7f3a7c2fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:02.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.820+0000 7f3a7c2fc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a74067a10 con 0x7f3a7413f160 2023-10-30T20:24:02.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.820+0000 7f3a7c2fc640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:02.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.820+0000 7f3a7c2fc640 10 monclient(hunting): authenticate will time out at 1566.109747s 2023-10-30T20:24:02.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.821+0000 7f3a7a899640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7413f160 0x7f3a74142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:02.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.821+0000 7f3a7a098640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3a740b4420 0x7f3a7413ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47800/0 (socket says 172.21.15.69:47800) 2023-10-30T20:24:02.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.821+0000 7f3a7a098640 1 -- 172.21.15.69:0/385976192 learned_addr learned my addr 172.21.15.69:0/385976192 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:24:02.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.821+0000 7f3a7a899640 10 monclient(hunting): get_auth_request con 0x7f3a7413f160 auth_method 0 2023-10-30T20:24:02.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.822+0000 7f3a7a899640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:02.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.822+0000 7f3a7a899640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:02.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.822+0000 7f3a7a899640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:02.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.822+0000 7f3a7a899640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:02.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.822+0000 7f3a7a899640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:02.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.822+0000 7f3a7a899640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:02.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.824+0000 7f3a7a899640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2023-10-30T20:24:02.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.824+0000 7f3a7a899640 1 -- 172.21.15.69:0/385976192 >> v1:172.21.15.183:6789/0 conn(0x7f3a740b12d0 legacy=0x7f3a74141910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:02.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.824+0000 7f3a7a899640 1 -- 172.21.15.69:0/385976192 >> v1:172.21.15.69:6789/0 conn(0x7f3a740b4420 legacy=0x7f3a7413ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:02.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.824+0000 7f3a7a899640 10 monclient: _finish_hunting 0 2023-10-30T20:24:02.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.824+0000 7f3a7a899640 1 monclient: found mon.noname-b 2023-10-30T20:24:02.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.824+0000 7f3a7a899640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:02.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.824+0000 7f3a79096640 1 -- 172.21.15.69:0/385976192 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122914340 0 0) 0x7f3a64001110 con 0x7f3a740b12d0 2023-10-30T20:24:02.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.824+0000 7f3a7a899640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:24:02.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.824+0000 7f3a7a899640 1 -- 172.21.15.69:0/385976192 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a74142560 con 0x7f3a7413f160 2023-10-30T20:24:02.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.824+0000 7f3a79096640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T20:24:02.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.826+0000 7f3a7a899640 1 --2- 172.21.15.69:0/385976192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7413f160 0x7f3a74142020 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3a700097d0 tx=0x7f3a7002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b0385d97c8450ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:02.832 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a79096640 1 -- 172.21.15.69:0/385976192 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a70031800 con 0x7f3a7413f160 2023-10-30T20:24:02.833 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a79096640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:02.833 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a79096640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:02.833 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a79096640 10 monclient: dump: 2023-10-30T20:24:02.833 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:02.833 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:02.834 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:02.834 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:02.834 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:02.834 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:02.834 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:02.834 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:02.835 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:02.835 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:02.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a79096640 10 monclient: _finish_auth 0 2023-10-30T20:24:02.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a79096640 10 monclient: _check_auth_tickets 2023-10-30T20:24:02.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a79096640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:02.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a79096640 1 -- 172.21.15.69:0/385976192 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a70031970 con 0x7f3a7413f160 2023-10-30T20:24:02.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a7c2fc640 5 monclient: authenticate success, global_id 4229 2023-10-30T20:24:02.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a7c2fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:02.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a79096640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:02.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a7c2fc640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:02.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a79096640 1 -- 172.21.15.69:0/385976192 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a70031d10 con 0x7f3a7413f160 2023-10-30T20:24:02.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a7c2fc640 4 set_mon_vals no callback set 2023-10-30T20:24:02.837 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a79096640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:02.837 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a79096640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:02.837 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a79096640 10 monclient: dump: 2023-10-30T20:24:02.837 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:02.837 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:02.838 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:02.838 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:02.838 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:02.838 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:02.838 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:02.838 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:02.839 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:02.839 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:02.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a7c2fc640 10 monclient: shutdown 2023-10-30T20:24:02.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a7c2fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:02.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.830+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/385976192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7413f160 msgr2=0x7f3a74142020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:02.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.831+0000 7f3a7c2fc640 1 --2- 172.21.15.69:0/385976192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7413f160 0x7f3a74142020 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3a700097d0 tx=0x7f3a7002b8a0 comp rx=0 tx=0).stop 2023-10-30T20:24:02.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.831+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/385976192 shutdown_connections 2023-10-30T20:24:02.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.831+0000 7f3a7c2fc640 1 --2- 172.21.15.69:0/385976192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7413f160 0x7f3a74142020 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:02.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.831+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/385976192 >> 172.21.15.69:0/385976192 conn(0x7f3a74065ea0 msgr2=0x7f3a740662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:02.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.831+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/385976192 shutdown_connections 2023-10-30T20:24:02.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.831+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/385976192 wait complete. 2023-10-30T20:24:02.843 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.832+0000 7f3a7c2fc640 10 monclient: build_initial_monmap 2023-10-30T20:24:02.843 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.832+0000 7f3a7c2fc640 10 monclient: monmap: 2023-10-30T20:24:02.843 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:02.843 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:02.844 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:02.844 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:02.844 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:02.844 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:02.844 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:02.844 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:02.845 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.832+0000 7f3a7c2fc640 1 librados: starting msgr at 2023-10-30T20:24:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.832+0000 7f3a7c2fc640 1 librados: starting objecter 2023-10-30T20:24:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.832+0000 7f3a7c2fc640 1 Processor -- start 2023-10-30T20:24:02.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.832+0000 7f3a7c2fc640 1 -- start start 2023-10-30T20:24:02.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.832+0000 7f3a7c2fc640 1 librados: setting wanted keys 2023-10-30T20:24:02.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.832+0000 7f3a7c2fc640 1 librados: calling monclient init 2023-10-30T20:24:02.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.832+0000 7f3a7c2fc640 10 monclient: init 2023-10-30T20:24:02.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:02.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:24:02.847 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a740b12d0 0x7f3a74138150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:02.847 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 10 monclient(hunting): picked mon.noname-a con 0x7f3a740b12d0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:02.847 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a740b4420 0x7f3a741ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:02.847 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 10 monclient(hunting): picked mon.noname-b con 0x7f3a740b4420 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:02.847 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7413f160 0x7f3a74137530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:02.847 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 10 monclient(hunting): picked mon.noname-c con 0x7f3a7413f160 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:02.848 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:02.848 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a740b3230 con 0x7f3a740b12d0 2023-10-30T20:24:02.848 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:02.848 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a741413e0 con 0x7f3a740b4420 2023-10-30T20:24:02.848 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:02.848 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a740b7070 con 0x7f3a7413f160 2023-10-30T20:24:02.849 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:02.849 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.833+0000 7f3a7c2fc640 10 monclient(hunting): authenticate will time out at 1566.122555s 2023-10-30T20:24:02.849 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.834+0000 7f3a7a899640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7413f160 0x7f3a74137530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:02.849 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.834+0000 7f3a79897640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a740b4420 0x7f3a741ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:02.849 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.834+0000 7f3a7a899640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7413f160 0x7f3a74137530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60558/0 (socket says 172.21.15.69:60558) 2023-10-30T20:24:02.850 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.834+0000 7f3a7a899640 1 -- 172.21.15.69:0/2443289638 learned_addr learned my addr 172.21.15.69:0/2443289638 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:02.850 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.834+0000 7f3a7a899640 10 monclient(hunting): get_auth_request con 0x7f3a7413f160 auth_method 0 2023-10-30T20:24:02.850 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.834+0000 7f3a7a899640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:02.850 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.834+0000 7f3a7a899640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:02.850 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.834+0000 7f3a7a899640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:02.850 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.834+0000 7f3a79897640 10 monclient(hunting): get_auth_request con 0x7f3a740b4420 auth_method 0 2023-10-30T20:24:02.851 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.834+0000 7f3a79897640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:02.851 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.834+0000 7f3a79897640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:02.851 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.834+0000 7f3a79897640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:02.851 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.835+0000 7f3a7a899640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:02.851 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.835+0000 7f3a7a899640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:02.851 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.835+0000 7f3a7a899640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:02.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.835+0000 7f3a79897640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:02.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.835+0000 7f3a79897640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:02.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.835+0000 7f3a79897640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:02.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.837+0000 7f3a7a899640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2023-10-30T20:24:02.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.837+0000 7f3a7a899640 1 -- 172.21.15.69:0/2443289638 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a740b4420 msgr2=0x7f3a741ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:02.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.837+0000 7f3a7a899640 1 --2- 172.21.15.69:0/2443289638 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a740b4420 0x7f3a741ca130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:02.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.837+0000 7f3a7a899640 1 -- 172.21.15.69:0/2443289638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a740b12d0 msgr2=0x7f3a74138150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:02.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.837+0000 7f3a7a899640 1 --2- 172.21.15.69:0/2443289638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a740b12d0 0x7f3a74138150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:02.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.837+0000 7f3a7a899640 10 monclient: _finish_hunting 0 2023-10-30T20:24:02.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.837+0000 7f3a7a899640 1 monclient: found mon.noname-c 2023-10-30T20:24:02.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.837+0000 7f3a7a899640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:02.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.837+0000 7f3a7a899640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:24:02.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.837+0000 7f3a7a899640 1 -- 172.21.15.69:0/2443289638 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a740b54d0 con 0x7f3a7413f160 2023-10-30T20:24:02.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.840+0000 7f3a7a899640 1 --2- 172.21.15.69:0/2443289638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7413f160 0x7f3a74137530 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3a700097d0 tx=0x7f3a700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:02.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.840+0000 7f3a6affd640 10 client.?.objecter ms_handle_connect 0x7f3a7413f160 2023-10-30T20:24:02.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.840+0000 7f3a6affd640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:02.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a6affd640 1 -- 172.21.15.69:0/2443289638 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7003acb0 con 0x7f3a7413f160 2023-10-30T20:24:02.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:02.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a6affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:02.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a6affd640 10 monclient: dump: 2023-10-30T20:24:02.857 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:02.857 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:02.857 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:02.857 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:02.857 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:02.857 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:02.858 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a6affd640 10 monclient: _finish_auth 0 2023-10-30T20:24:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a6affd640 10 monclient: _check_auth_tickets 2023-10-30T20:24:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a7c2fc640 5 monclient: authenticate success, global_id 4232 2023-10-30T20:24:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a6affd640 1 -- 172.21.15.69:0/2443289638 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7003e030 con 0x7f3a7413f160 2023-10-30T20:24:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a6affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a6affd640 1 -- 172.21.15.69:0/2443289638 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a70044630 con 0x7f3a7413f160 2023-10-30T20:24:02.860 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:02.860 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:02.860 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a6affd640 10 monclient: dump: 2023-10-30T20:24:02.860 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:02.860 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:02.862 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a6bfff640 4 set_mon_vals no callback set 2023-10-30T20:24:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a7c2fc640 10 monclient: _renew_subs 2023-10-30T20:24:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a7c2fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/2443289638 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a740b4fe0 con 0x7f3a7413f160 2023-10-30T20:24:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a7c2fc640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a7c2fc640 10 monclient: _renew_subs 2023-10-30T20:24:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a7c2fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/2443289638 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a74137a70 con 0x7f3a7413f160 2023-10-30T20:24:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.846+0000 7f3a7c2fc640 1 librados: init done 2023-10-30T20:24:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.847+0000 7f3a7c2fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.847+0000 7f3a7c2fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.847+0000 7f3a7c2fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.847+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/2443289638 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a740753b0 con 0x7f3a7413f160 2023-10-30T20:24:02.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.867+0000 7f3a6affd640 1 -- 172.21.15.69:0/2443289638 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f3a7004d450 con 0x7f3a7413f160 2023-10-30T20:24:02.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.867+0000 7f3a6affd640 1 --2- 172.21.15.69:0/2443289638 >> v2:172.21.15.69:6800/1876307353 conn(0x7f3a5806c810 0x7f3a5806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.867+0000 7f3a7a098640 1 --2- 172.21.15.69:0/2443289638 >> v2:172.21.15.69:6800/1876307353 conn(0x7f3a5806c810 0x7f3a5806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.867+0000 7f3a7a098640 10 monclient: get_auth_request con 0x7f3a5806c810 auth_method 0 2023-10-30T20:24:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.868+0000 7f3a6affd640 10 client.4232.objecter ms_handle_connect 0x7f3a5806c810 2023-10-30T20:24:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:02.868+0000 7f3a7a098640 1 --2- 172.21.15.69:0/2443289638 >> v2:172.21.15.69:6800/1876307353 conn(0x7f3a5806c810 0x7f3a5806ecd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0012e0 tx=0x7f3a6c001630 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:03.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:03.111+0000 7f3a6affd640 1 -- 172.21.15.69:0/2443289638 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4079+0+0 (secure 0 0 0) 0x7f3a70052080 con 0x7f3a7413f160 2023-10-30T20:24:03.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:03.111+0000 7f3a6affd640 10 client.4232.objecter ms_dispatch 0x7f3a74000b90 osd_map(19..19 src has 1..19) v4 2023-10-30T20:24:03.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:03.111+0000 7f3a6affd640 3 client.4232.objecter handle_osd_map got epochs [19,19] > 0 2023-10-30T20:24:03.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:03.111+0000 7f3a6affd640 3 client.4232.objecter handle_osd_map decoding full epoch 19 2023-10-30T20:24:03.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:03.111+0000 7f3a6affd640 20 client.4232.objecter dump_active .. 0 homeless 2023-10-30T20:24:03.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:03.111+0000 7f3a6affd640 1 -- 172.21.15.69:0/2443289638 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3a70031920 con 0x7f3a7413f160 2023-10-30T20:24:03.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:03.111+0000 7f3a6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:03.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:03.111+0000 7f3a6affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:03.244 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:25.302 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:03.314 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:25.372 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:03.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:03.339+0000 7f3a7c2fc640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T20:24:03.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:03.339+0000 7f3a7c2fc640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T20:24:03.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:03.339+0000 7f3a7c2fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:03.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:03.339+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/2443289638 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3a74142560 con 0x7f3a7413f160 2023-10-30T20:24:03.342 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:25.401 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:03.354 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:25.381 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:03.408 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:25.435 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:03.423 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:25.482 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:03.580 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:25.639 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:03.649 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:25.676 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:03.679 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:25.738 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:04.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.024+0000 7f3a6affd640 1 -- 172.21.15.69:0/2443289638 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v20) v1 ==== 173+0+0 (secure 0 0 0) 0x7f3a7007a7d0 con 0x7f3a7413f160 2023-10-30T20:24:04.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.024+0000 7f3a6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T20:24:04.026 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.024+0000 7f3a6affd640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-10-30T20:24:04.026 INFO:teuthology.orchestra.run.smithi069.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-30T20:24:04.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 10 librados: watch_flush enter 2023-10-30T20:24:04.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 10 librados: watch_flush exit 2023-10-30T20:24:04.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 20 client.4232.objecter shutdown clearing up homeless session... 2023-10-30T20:24:04.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 10 client.4232.objecter successfully canceled tick 2023-10-30T20:24:04.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/2443289638 >> v2:172.21.15.69:6800/1876307353 conn(0x7f3a5806c810 msgr2=0x7f3a5806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:04.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 1 --2- 172.21.15.69:0/2443289638 >> v2:172.21.15.69:6800/1876307353 conn(0x7f3a5806c810 0x7f3a5806ecd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0012e0 tx=0x7f3a6c001630 comp rx=0 tx=0).stop 2023-10-30T20:24:04.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 10 monclient: shutdown 2023-10-30T20:24:04.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:04.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/2443289638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7413f160 msgr2=0x7f3a74137530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:04.030 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 1 --2- 172.21.15.69:0/2443289638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7413f160 0x7f3a74137530 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3a700097d0 tx=0x7f3a700046c0 comp rx=0 tx=0).stop 2023-10-30T20:24:04.030 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/2443289638 shutdown_connections 2023-10-30T20:24:04.030 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 1 --2- 172.21.15.69:0/2443289638 >> v2:172.21.15.69:6800/1876307353 conn(0x7f3a5806c810 0x7f3a5806ecd0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:04.030 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 1 --2- 172.21.15.69:0/2443289638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7413f160 0x7f3a74137530 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:04.030 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 1 --2- 172.21.15.69:0/2443289638 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a740b4420 0x7f3a741ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:04.031 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 1 --2- 172.21.15.69:0/2443289638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a740b12d0 0x7f3a74138150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:04.031 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/2443289638 >> 172.21.15.69:0/2443289638 conn(0x7f3a74065ea0 msgr2=0x7f3a7413dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:04.031 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/2443289638 shutdown_connections 2023-10-30T20:24:04.031 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 1 -- 172.21.15.69:0/2443289638 wait complete. 2023-10-30T20:24:04.031 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.026+0000 7f3a7c2fc640 1 librados: shutdown 2023-10-30T20:24:04.038 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-30T20:24:04.039 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-30T20:24:04.039 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-30T20:24:04.040 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-30T20:24:04.040 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-30T20:24:04.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.144+0000 7fc67e4fc640 10 monclient: get_monmap_and_config 2023-10-30T20:24:04.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.144+0000 7fc67e4fc640 10 monclient: build_initial_monmap 2023-10-30T20:24:04.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.144+0000 7fc67e4fc640 10 monclient: monmap: 2023-10-30T20:24:04.146 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:04.146 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:04.146 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:04.145496+0000 2023-10-30T20:24:04.147 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:04.145496+0000 2023-10-30T20:24:04.147 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:04.147 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:04.147 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:04.147 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:04.148 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:04.148 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:04.148 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:04.148 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:04.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 1 Processor -- start 2023-10-30T20:24:04.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 1 -- start start 2023-10-30T20:24:04.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 10 monclient: init 2023-10-30T20:24:04.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:04.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-10-30T20:24:04.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6780b4390 0x7fc6780a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:04.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 10 monclient(hunting): picked mon.noname-b con 0x7fc6780b4390 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:04.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6781418a0 addr v1:172.21.15.69:6789/0 2023-10-30T20:24:04.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc678141ce0 0x7fc6780b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:04.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 10 monclient(hunting): picked mon.noname-a con 0x7fc678141ce0 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:04.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6780a8f60 con 0x7fc6781418a0 2023-10-30T20:24:04.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:04.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc678067a10 con 0x7fc678141ce0 2023-10-30T20:24:04.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:04.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6780b4770 con 0x7fc6780b4390 2023-10-30T20:24:04.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:04.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.145+0000 7fc67e4fc640 10 monclient(hunting): authenticate will time out at 1567.434734s 2023-10-30T20:24:04.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.146+0000 7fc67ca99640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc678141ce0 0x7fc6780b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:04.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.146+0000 7fc677fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6780b4390 0x7fc6780a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:04.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.146+0000 7fc677fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6780b4390 0x7fc6780a8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60568/0 (socket says 172.21.15.69:60568) 2023-10-30T20:24:04.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.146+0000 7fc677fff640 1 -- 172.21.15.69:0/2692769000 learned_addr learned my addr 172.21.15.69:0/2692769000 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:04.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.146+0000 7fc677fff640 10 monclient(hunting): get_auth_request con 0x7fc6780b4390 auth_method 0 2023-10-30T20:24:04.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.146+0000 7fc677fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:04.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.146+0000 7fc677fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:04.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.146+0000 7fc677fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:04.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.146+0000 7fc67ca99640 10 monclient(hunting): get_auth_request con 0x7fc678141ce0 auth_method 0 2023-10-30T20:24:04.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.146+0000 7fc67ca99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:04.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.146+0000 7fc67ca99640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:04.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.146+0000 7fc67ca99640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:04.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.147+0000 7fc677fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:04.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.147+0000 7fc677fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:04.157 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.147+0000 7fc677fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:04.157 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.147+0000 7fc67ca99640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:04.157 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.147+0000 7fc67ca99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:04.157 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.147+0000 7fc67ca99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:04.157 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.149+0000 7fc67ca99640 10 monclient(hunting): handle_auth_done global_id 4260 payload 274 2023-10-30T20:24:04.157 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.149+0000 7fc67ca99640 1 -- 172.21.15.69:0/2692769000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6780b4390 msgr2=0x7fc6780a8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:04.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.149+0000 7fc67ca99640 1 --2- 172.21.15.69:0/2692769000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6780b4390 0x7fc6780a8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:04.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.149+0000 7fc67ca99640 1 -- 172.21.15.69:0/2692769000 >> v1:172.21.15.69:6789/0 conn(0x7fc6781418a0 legacy=0x7fc6780b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:04.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.149+0000 7fc67ca99640 10 monclient: _finish_hunting 0 2023-10-30T20:24:04.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.149+0000 7fc67ca99640 1 monclient: found mon.noname-a 2023-10-30T20:24:04.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.149+0000 7fc67ca99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:04.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.149+0000 7fc67ca99640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:04.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.149+0000 7fc67ca99640 1 -- 172.21.15.69:0/2692769000 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc678142170 con 0x7fc678141ce0 2023-10-30T20:24:04.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.152+0000 7fc67ca99640 1 --2- 172.21.15.69:0/2692769000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc678141ce0 0x7fc6780b2210 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc670009d90 tx=0x7fc67002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca1d2c275a2cf85 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:04.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.155+0000 7fc676ffd640 1 -- 172.21.15.69:0/2692769000 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67002cc50 con 0x7fc678141ce0 2023-10-30T20:24:04.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.155+0000 7fc676ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:04.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.155+0000 7fc676ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:04.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.155+0000 7fc676ffd640 10 monclient: dump: 2023-10-30T20:24:04.163 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:04.164 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:04.164 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:04.164 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:04.164 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:04.164 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:04.165 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:04.165 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:04.165 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:04.165 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:04.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.155+0000 7fc676ffd640 10 monclient: _finish_auth 0 2023-10-30T20:24:04.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.155+0000 7fc676ffd640 10 monclient: _check_auth_tickets 2023-10-30T20:24:04.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.155+0000 7fc676ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:04.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.155+0000 7fc67e4fc640 5 monclient: authenticate success, global_id 4260 2023-10-30T20:24:04.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.155+0000 7fc67e4fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:04.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.157+0000 7fc676ffd640 1 -- 172.21.15.69:0/2692769000 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67002cdc0 con 0x7fc678141ce0 2023-10-30T20:24:04.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.157+0000 7fc676ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:04.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.157+0000 7fc676ffd640 1 -- 172.21.15.69:0/2692769000 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670034660 con 0x7fc678141ce0 2023-10-30T20:24:04.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.157+0000 7fc67e4fc640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:04.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.157+0000 7fc67e4fc640 4 set_mon_vals no callback set 2023-10-30T20:24:04.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.157+0000 7fc676ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:04.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.157+0000 7fc676ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:04.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.157+0000 7fc676ffd640 10 monclient: dump: 2023-10-30T20:24:04.168 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:04.168 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:04.168 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:04.168 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:04.168 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:04.169 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:04.169 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:04.169 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:04.169 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:04.169 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:04.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.158+0000 7fc67e4fc640 10 monclient: shutdown 2023-10-30T20:24:04.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.158+0000 7fc67e4fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:04.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.158+0000 7fc67e4fc640 1 -- 172.21.15.69:0/2692769000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc678141ce0 msgr2=0x7fc6780b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:04.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.158+0000 7fc67e4fc640 1 --2- 172.21.15.69:0/2692769000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc678141ce0 0x7fc6780b2210 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc670009d90 tx=0x7fc67002bea0 comp rx=0 tx=0).stop 2023-10-30T20:24:04.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.158+0000 7fc67e4fc640 1 -- 172.21.15.69:0/2692769000 shutdown_connections 2023-10-30T20:24:04.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.158+0000 7fc67e4fc640 1 --2- 172.21.15.69:0/2692769000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc678141ce0 0x7fc6780b2210 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:04.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.158+0000 7fc67e4fc640 1 --2- 172.21.15.69:0/2692769000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6780b4390 0x7fc6780a8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:04.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.158+0000 7fc67e4fc640 1 -- 172.21.15.69:0/2692769000 >> 172.21.15.69:0/2692769000 conn(0x7fc678065ea0 msgr2=0x7fc6780662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:04.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.158+0000 7fc67e4fc640 1 -- 172.21.15.69:0/2692769000 shutdown_connections 2023-10-30T20:24:04.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.158+0000 7fc67e4fc640 1 -- 172.21.15.69:0/2692769000 wait complete. 2023-10-30T20:24:04.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.158+0000 7fc67e4fc640 10 monclient: build_initial_monmap 2023-10-30T20:24:04.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.158+0000 7fc67e4fc640 10 monclient: monmap: 2023-10-30T20:24:04.172 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:04.172 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:04.172 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:04.172 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:04.173 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:04.173 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:04.173 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:04.173 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:04.173 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:04.174 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:04.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 1 librados: starting msgr at 2023-10-30T20:24:04.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 1 librados: starting objecter 2023-10-30T20:24:04.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 1 Processor -- start 2023-10-30T20:24:04.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 1 -- start start 2023-10-30T20:24:04.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 1 librados: setting wanted keys 2023-10-30T20:24:04.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 1 librados: calling monclient init 2023-10-30T20:24:04.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 10 monclient: init 2023-10-30T20:24:04.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:04.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T20:24:04.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6780b4390 0x7fc6780ab880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:04.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 10 monclient(hunting): picked mon.noname-c con 0x7fc6780b4390 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:04.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6781418a0 0x7fc6781da5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:04.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 10 monclient(hunting): picked mon.noname-b con 0x7fc6781418a0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:04.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc678141ce0 0x7fc6780b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:04.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 10 monclient(hunting): picked mon.noname-a con 0x7fc678141ce0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:04.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:04.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6780b11d0 con 0x7fc678141ce0 2023-10-30T20:24:04.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:04.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc678139110 con 0x7fc6781418a0 2023-10-30T20:24:04.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:04.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6781da0b0 con 0x7fc6780b4390 2023-10-30T20:24:04.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:04.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.159+0000 7fc67e4fc640 10 monclient(hunting): authenticate will time out at 1567.448772s 2023-10-30T20:24:04.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.160+0000 7fc67ca99640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc678141ce0 0x7fc6780b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:04.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.160+0000 7fc677fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6780b4390 0x7fc6780ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:04.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.160+0000 7fc6777fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6781418a0 0x7fc6781da5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:04.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.160+0000 7fc67ca99640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc678141ce0 0x7fc6780b0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45146/0 (socket says 172.21.15.69:45146) 2023-10-30T20:24:04.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.160+0000 7fc67ca99640 1 -- 172.21.15.69:0/565481661 learned_addr learned my addr 172.21.15.69:0/565481661 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:04.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.160+0000 7fc67ca99640 10 monclient(hunting): get_auth_request con 0x7fc678141ce0 auth_method 0 2023-10-30T20:24:04.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.160+0000 7fc67ca99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:04.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.160+0000 7fc67ca99640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:04.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.160+0000 7fc67ca99640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:04.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc6777fe640 10 monclient(hunting): get_auth_request con 0x7fc6781418a0 auth_method 0 2023-10-30T20:24:04.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc6777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:04.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc6777fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:04.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc6777fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:04.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc677fff640 10 monclient(hunting): get_auth_request con 0x7fc6780b4390 auth_method 0 2023-10-30T20:24:04.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc677fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:04.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc677fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:04.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc677fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:04.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc67ca99640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:04.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc67ca99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:04.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc67ca99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:04.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc6777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:04.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc6777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:04.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc6777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:04.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc677fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:04.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.161+0000 7fc677fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:04.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.162+0000 7fc677fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:04.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.164+0000 7fc67ca99640 10 monclient(hunting): handle_auth_done global_id 4263 payload 824 2023-10-30T20:24:04.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.164+0000 7fc67ca99640 1 -- 172.21.15.69:0/565481661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6780b4390 msgr2=0x7fc6780ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:04.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.164+0000 7fc67ca99640 1 --2- 172.21.15.69:0/565481661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6780b4390 0x7fc6780ab880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:04.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.164+0000 7fc67ca99640 1 -- 172.21.15.69:0/565481661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6781418a0 msgr2=0x7fc6781da5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:04.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.164+0000 7fc67ca99640 1 --2- 172.21.15.69:0/565481661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6781418a0 0x7fc6781da5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:04.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.164+0000 7fc67ca99640 10 monclient: _finish_hunting 0 2023-10-30T20:24:04.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.164+0000 7fc67ca99640 1 monclient: found mon.noname-a 2023-10-30T20:24:04.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.164+0000 7fc67ca99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:04.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.164+0000 7fc67ca99640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:04.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.164+0000 7fc67ca99640 1 -- 172.21.15.69:0/565481661 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc678143f40 con 0x7fc678141ce0 2023-10-30T20:24:04.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.167+0000 7fc67ca99640 1 --2- 172.21.15.69:0/565481661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc678141ce0 0x7fc6780b0610 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc670009eb0 tx=0x7fc670004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:04.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.167+0000 7fc674ff9640 10 client.?.objecter ms_handle_connect 0x7fc678141ce0 2023-10-30T20:24:04.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.167+0000 7fc674ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:04.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc674ff9640 1 -- 172.21.15.69:0/565481661 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67003e610 con 0x7fc678141ce0 2023-10-30T20:24:04.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc674ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:04.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc674ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:04.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc674ff9640 10 monclient: dump: 2023-10-30T20:24:04.188 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:04.188 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:04.188 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:04.188 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:04.189 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:04.189 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:04.189 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:04.189 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:04.189 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:04.189 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:04.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc674ff9640 10 monclient: _finish_auth 0 2023-10-30T20:24:04.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc674ff9640 10 monclient: _check_auth_tickets 2023-10-30T20:24:04.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc674ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:04.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc674ff9640 1 -- 172.21.15.69:0/565481661 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67003e780 con 0x7fc678141ce0 2023-10-30T20:24:04.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc67e4fc640 5 monclient: authenticate success, global_id 4263 2023-10-30T20:24:04.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc67e4fc640 10 monclient: _renew_subs 2023-10-30T20:24:04.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc67e4fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:04.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc67e4fc640 1 -- 172.21.15.69:0/565481661 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc678144140 con 0x7fc678141ce0 2023-10-30T20:24:04.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc674ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:04.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc674ff9640 1 -- 172.21.15.69:0/565481661 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67003eb40 con 0x7fc678141ce0 2023-10-30T20:24:04.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc674ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:04.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc674ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:04.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc674ff9640 10 monclient: dump: 2023-10-30T20:24:04.192 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:04.192 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:04.192 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:04.193 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:04.193 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:04.193 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:04.193 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:04.193 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc67e4fc640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc6767fc640 4 set_mon_vals no callback set 2023-10-30T20:24:04.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc67e4fc640 10 monclient: _renew_subs 2023-10-30T20:24:04.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc67e4fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:04.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc67e4fc640 1 -- 172.21.15.69:0/565481661 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6781dacf0 con 0x7fc678141ce0 2023-10-30T20:24:04.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.172+0000 7fc67e4fc640 1 librados: init done 2023-10-30T20:24:04.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.174+0000 7fc67e4fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:04.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.174+0000 7fc67e4fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:04.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.174+0000 7fc67e4fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:04.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.174+0000 7fc67e4fc640 1 -- 172.21.15.69:0/565481661 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6780a9900 con 0x7fc678141ce0 2023-10-30T20:24:04.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.195+0000 7fc674ff9640 1 -- 172.21.15.69:0/565481661 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fc67003b050 con 0x7fc678141ce0 2023-10-30T20:24:04.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.195+0000 7fc674ff9640 1 --2- 172.21.15.69:0/565481661 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc66406c7c0 0x7fc66406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:04.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.196+0000 7fc677fff640 1 --2- 172.21.15.69:0/565481661 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc66406c7c0 0x7fc66406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:04.198 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.196+0000 7fc677fff640 10 monclient: get_auth_request con 0x7fc66406c7c0 auth_method 0 2023-10-30T20:24:04.198 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.196+0000 7fc674ff9640 10 client.4263.objecter ms_handle_connect 0x7fc66406c7c0 2023-10-30T20:24:04.198 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.196+0000 7fc677fff640 1 --2- 172.21.15.69:0/565481661 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc66406c7c0 0x7fc66406ec80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc668009bd0 tx=0x7fc668009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:04.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.202+0000 7fc674ff9640 1 -- 172.21.15.69:0/565481661 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fc670079fb0 con 0x7fc678141ce0 2023-10-30T20:24:04.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.202+0000 7fc674ff9640 10 client.4263.objecter ms_dispatch 0x7fc678000b90 osd_map(20..20 src has 1..20) v4 2023-10-30T20:24:04.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.202+0000 7fc674ff9640 3 client.4263.objecter handle_osd_map got epochs [20,20] > 0 2023-10-30T20:24:04.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.202+0000 7fc674ff9640 3 client.4263.objecter handle_osd_map decoding full epoch 20 2023-10-30T20:24:04.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.202+0000 7fc674ff9640 20 client.4263.objecter dump_active .. 0 homeless 2023-10-30T20:24:04.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.485+0000 7fc674ff9640 1 -- 172.21.15.69:0/565481661 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc67007e050 con 0x7fc678141ce0 2023-10-30T20:24:04.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.485+0000 7fc674ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:04.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.485+0000 7fc674ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:04.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.766+0000 7fc67e4fc640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-10-30T20:24:04.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.766+0000 7fc67e4fc640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-10-30T20:24:04.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.766+0000 7fc67e4fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:04.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:04.766+0000 7fc67e4fc640 1 -- 172.21.15.69:0/565481661 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fc6781d9f10 con 0x7fc678141ce0 2023-10-30T20:24:05.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.140+0000 7fc674ff9640 1 -- 172.21.15.69:0/565481661 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170969 (secure 0 0 0) 0x7fc670082200 con 0x7fc678141ce0 2023-10-30T20:24:05.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.140+0000 7fc674ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-10-30T20:24:05.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.140+0000 7fc674ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:05.144 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:24:05.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 10 librados: watch_flush enter 2023-10-30T20:24:05.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 10 librados: watch_flush exit 2023-10-30T20:24:05.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 20 client.4263.objecter shutdown clearing up homeless session... 2023-10-30T20:24:05.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 10 client.4263.objecter successfully canceled tick 2023-10-30T20:24:05.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 1 -- 172.21.15.69:0/565481661 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc66406c7c0 msgr2=0x7fc66406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:05.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 1 --2- 172.21.15.69:0/565481661 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc66406c7c0 0x7fc66406ec80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc668009bd0 tx=0x7fc668009240 comp rx=0 tx=0).stop 2023-10-30T20:24:05.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 10 monclient: shutdown 2023-10-30T20:24:05.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:05.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 1 -- 172.21.15.69:0/565481661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc678141ce0 msgr2=0x7fc6780b0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:05.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 1 --2- 172.21.15.69:0/565481661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc678141ce0 0x7fc6780b0610 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc670009eb0 tx=0x7fc670004a40 comp rx=0 tx=0).stop 2023-10-30T20:24:05.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 1 -- 172.21.15.69:0/565481661 shutdown_connections 2023-10-30T20:24:05.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 1 --2- 172.21.15.69:0/565481661 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc66406c7c0 0x7fc66406ec80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:05.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 1 --2- 172.21.15.69:0/565481661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc678141ce0 0x7fc6780b0610 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:05.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 1 --2- 172.21.15.69:0/565481661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6781418a0 0x7fc6781da5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:05.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 1 --2- 172.21.15.69:0/565481661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6780b4390 0x7fc6780ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:05.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 1 -- 172.21.15.69:0/565481661 >> 172.21.15.69:0/565481661 conn(0x7fc678065ea0 msgr2=0x7fc6780b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:05.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 1 -- 172.21.15.69:0/565481661 shutdown_connections 2023-10-30T20:24:05.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.143+0000 7fc67e4fc640 1 -- 172.21.15.69:0/565481661 wait complete. 2023-10-30T20:24:05.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.144+0000 7fc67e4fc640 1 librados: shutdown 2023-10-30T20:24:05.157 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":5,"active_gid":4105,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1876307353}]},"active_addr":"172.21.15.69:6800/1876307353","active_change":"2023-10-30T20:23:11.989291+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4100,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":506945381}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":3956698573}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":3139805683}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":1569079362}]}]} 2023-10-30T20:24:05.163 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-30T20:24:05.163 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-30T20:24:05.163 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T20:24:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.269+0000 7fa777d90640 10 monclient: get_monmap_and_config 2023-10-30T20:24:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.269+0000 7fa777d90640 10 monclient: build_initial_monmap 2023-10-30T20:24:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.269+0000 7fa777d90640 10 monclient: monmap: 2023-10-30T20:24:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:05.270174+0000 2023-10-30T20:24:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:05.270174+0000 2023-10-30T20:24:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:05.273 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:05.273 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:05.273 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:05.273 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:05.273 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:05.274 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:05.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.269+0000 7fa777d90640 1 Processor -- start 2023-10-30T20:24:05.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.269+0000 7fa777d90640 1 -- start start 2023-10-30T20:24:05.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.269+0000 7fa777d90640 10 monclient: init 2023-10-30T20:24:05.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.270+0000 7fa777d90640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:05.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.270+0000 7fa777d90640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-10-30T20:24:05.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.270+0000 7fa777d90640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7700b4390 0x7fa7700a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.270+0000 7fa777d90640 10 monclient(hunting): picked mon.noname-b con 0x7fa7700b4390 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.270+0000 7fa777d90640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa7701418a0 0x7fa770141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.270+0000 7fa777d90640 10 monclient(hunting): picked mon.noname-c con 0x7fa7701418a0 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.270+0000 7fa777d90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7700b1b00 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.270+0000 7fa777d90640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7700a8f60 con 0x7fa7700b1b00 2023-10-30T20:24:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.270+0000 7fa777d90640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:05.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.270+0000 7fa777d90640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa770067a10 con 0x7fa7701418a0 2023-10-30T20:24:05.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.270+0000 7fa777d90640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:05.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.270+0000 7fa777d90640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7700b4770 con 0x7fa7700b4390 2023-10-30T20:24:05.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.270+0000 7fa777d90640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.270+0000 7fa777d90640 10 monclient(hunting): authenticate will time out at 1568.559367s 2023-10-30T20:24:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.271+0000 7fa775b2c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7700b4390 0x7fa7700a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.271+0000 7fa775b2c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7700b4390 0x7fa7700a8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60586/0 (socket says 172.21.15.69:60586) 2023-10-30T20:24:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.271+0000 7fa775b2c640 1 -- 172.21.15.69:0/346729250 learned_addr learned my addr 172.21.15.69:0/346729250 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.271+0000 7fa775b2c640 10 monclient(hunting): get_auth_request con 0x7fa7700b4390 auth_method 0 2023-10-30T20:24:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.271+0000 7fa775b2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.271+0000 7fa775b2c640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.271+0000 7fa775b2c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.271+0000 7fa77532b640 1 --2- 172.21.15.69:0/346729250 >> v2:172.21.15.183:3300/0 conn(0x7fa7701418a0 0x7fa770141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.272+0000 7fa775b2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.272+0000 7fa775b2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:05.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.272+0000 7fa775b2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:05.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.272+0000 7fa77532b640 10 monclient(hunting): get_auth_request con 0x7fa7701418a0 auth_method 0 2023-10-30T20:24:05.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.272+0000 7fa77532b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:05.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.272+0000 7fa77532b640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:05.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.272+0000 7fa77532b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.273+0000 7fa775b2c640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2023-10-30T20:24:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.273+0000 7fa775b2c640 1 -- 172.21.15.69:0/346729250 >> v2:172.21.15.183:3300/0 conn(0x7fa7701418a0 msgr2=0x7fa770141ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.273+0000 7fa775b2c640 1 --2- 172.21.15.69:0/346729250 >> v2:172.21.15.183:3300/0 conn(0x7fa7701418a0 0x7fa770141ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.273+0000 7fa775b2c640 1 -- 172.21.15.69:0/346729250 >> v1:172.21.15.183:6789/0 conn(0x7fa7700b1b00 legacy=0x7fa7700b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.273+0000 7fa775b2c640 10 monclient: _finish_hunting 0 2023-10-30T20:24:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.273+0000 7fa775b2c640 1 monclient: found mon.noname-b 2023-10-30T20:24:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.273+0000 7fa775b2c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.273+0000 7fa775b2c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:24:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.273+0000 7fa775b2c640 1 -- 172.21.15.69:0/346729250 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7700aa3d0 con 0x7fa7700b4390 2023-10-30T20:24:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.275+0000 7fa775b2c640 1 --2- 172.21.15.69:0/346729250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7700b4390 0x7fa7700a8770 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa760009870 tx=0x7fa76002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3364950ad3586e32 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.277+0000 7fa774b2a640 1 -- 172.21.15.69:0/346729250 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760030840 con 0x7fa7700b4390 2023-10-30T20:24:05.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.277+0000 7fa774b2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:05.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.277+0000 7fa774b2a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:05.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.277+0000 7fa774b2a640 10 monclient: dump: 2023-10-30T20:24:05.288 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:05.290 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:05.290 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:05.290 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:05.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.278+0000 7fa774b2a640 10 monclient: _finish_auth 0 2023-10-30T20:24:05.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.278+0000 7fa774b2a640 10 monclient: _check_auth_tickets 2023-10-30T20:24:05.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.278+0000 7fa774b2a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:05.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.278+0000 7fa777d90640 5 monclient: authenticate success, global_id 4241 2023-10-30T20:24:05.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.278+0000 7fa777d90640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:05.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.279+0000 7fa774b2a640 1 -- 172.21.15.69:0/346729250 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7600309b0 con 0x7fa7700b4390 2023-10-30T20:24:05.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.279+0000 7fa774b2a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:05.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.279+0000 7fa774b2a640 1 -- 172.21.15.69:0/346729250 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760030d50 con 0x7fa7700b4390 2023-10-30T20:24:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.279+0000 7fa777d90640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.279+0000 7fa777d90640 4 set_mon_vals no callback set 2023-10-30T20:24:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.279+0000 7fa774b2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.279+0000 7fa774b2a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.279+0000 7fa774b2a640 10 monclient: dump: 2023-10-30T20:24:05.293 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:05.293 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:05.293 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:05.293 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:05.294 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:05.294 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:05.294 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:05.294 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:05.294 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:05.295 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:05.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.280+0000 7fa777d90640 10 monclient: shutdown 2023-10-30T20:24:05.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.280+0000 7fa777d90640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:05.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.280+0000 7fa777d90640 1 -- 172.21.15.69:0/346729250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7700b4390 msgr2=0x7fa7700a8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:05.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.280+0000 7fa777d90640 1 --2- 172.21.15.69:0/346729250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7700b4390 0x7fa7700a8770 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa760009870 tx=0x7fa76002b980 comp rx=0 tx=0).stop 2023-10-30T20:24:05.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.280+0000 7fa777d90640 1 -- 172.21.15.69:0/346729250 shutdown_connections 2023-10-30T20:24:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.280+0000 7fa777d90640 1 --2- 172.21.15.69:0/346729250 >> v2:172.21.15.183:3300/0 conn(0x7fa7701418a0 0x7fa770141ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.280+0000 7fa777d90640 1 --2- 172.21.15.69:0/346729250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7700b4390 0x7fa7700a8770 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.280+0000 7fa777d90640 1 -- 172.21.15.69:0/346729250 >> 172.21.15.69:0/346729250 conn(0x7fa770065ea0 msgr2=0x7fa7700662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.280+0000 7fa777d90640 1 -- 172.21.15.69:0/346729250 shutdown_connections 2023-10-30T20:24:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.280+0000 7fa777d90640 1 -- 172.21.15.69:0/346729250 wait complete. 2023-10-30T20:24:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.281+0000 7fa777d90640 10 monclient: build_initial_monmap 2023-10-30T20:24:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.281+0000 7fa777d90640 10 monclient: monmap: 2023-10-30T20:24:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:05.299 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:05.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.281+0000 7fa777d90640 1 librados: starting msgr at 2023-10-30T20:24:05.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.281+0000 7fa777d90640 1 librados: starting objecter 2023-10-30T20:24:05.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.281+0000 7fa777d90640 1 Processor -- start 2023-10-30T20:24:05.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.281+0000 7fa777d90640 1 -- start start 2023-10-30T20:24:05.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.281+0000 7fa777d90640 1 librados: setting wanted keys 2023-10-30T20:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.281+0000 7fa777d90640 1 librados: calling monclient init 2023-10-30T20:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.281+0000 7fa777d90640 10 monclient: init 2023-10-30T20:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T20:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7700b1b00 0x7fa7700b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 10 monclient(hunting): picked mon.noname-a con 0x7fa7700b1b00 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7700b4390 0x7fa7700b6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 10 monclient(hunting): picked mon.noname-c con 0x7fa7700b4390 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa7701418a0 0x7fa7701da330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 10 monclient(hunting): picked mon.noname-b con 0x7fa7701418a0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7701424e0 con 0x7fa7700b1b00 2023-10-30T20:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa770143720 con 0x7fa7701418a0 2023-10-30T20:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7700b6c70 con 0x7fa7700b4390 2023-10-30T20:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa777d90640 10 monclient(hunting): authenticate will time out at 1568.571299s 2023-10-30T20:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.282+0000 7fa77532b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7700b4390 0x7fa7700b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa775b2c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7700b1b00 0x7fa7700b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa77532b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7700b4390 0x7fa7700b6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60588/0 (socket says 172.21.15.69:60588) 2023-10-30T20:24:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa77532b640 1 -- 172.21.15.69:0/2563563010 learned_addr learned my addr 172.21.15.69:0/2563563010 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa77532b640 10 monclient(hunting): get_auth_request con 0x7fa7700b4390 auth_method 0 2023-10-30T20:24:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa77532b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa77532b640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa77532b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:05.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa77632d640 1 --2- 172.21.15.69:0/2563563010 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa7701418a0 0x7fa7701da330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:05.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa775b2c640 10 monclient(hunting): get_auth_request con 0x7fa7700b1b00 auth_method 0 2023-10-30T20:24:05.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa775b2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:05.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa775b2c640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:05.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa775b2c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:05.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa77532b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:05.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa77532b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:05.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa77532b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:05.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa77632d640 10 monclient(hunting): get_auth_request con 0x7fa7701418a0 auth_method 0 2023-10-30T20:24:05.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa77632d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:05.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa77632d640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:05.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.283+0000 7fa77632d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:05.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.284+0000 7fa775b2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:05.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.284+0000 7fa775b2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:05.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.284+0000 7fa775b2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.284+0000 7fa77632d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.284+0000 7fa77632d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.284+0000 7fa77632d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.286+0000 7fa77532b640 10 monclient(hunting): handle_auth_done global_id 4244 payload 824 2023-10-30T20:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.286+0000 7fa77532b640 1 -- 172.21.15.69:0/2563563010 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa7701418a0 msgr2=0x7fa7701da330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.286+0000 7fa77532b640 1 --2- 172.21.15.69:0/2563563010 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa7701418a0 0x7fa7701da330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.286+0000 7fa77532b640 1 -- 172.21.15.69:0/2563563010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7700b1b00 msgr2=0x7fa7700b5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.286+0000 7fa77532b640 1 --2- 172.21.15.69:0/2563563010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7700b1b00 0x7fa7700b5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.286+0000 7fa77532b640 10 monclient: _finish_hunting 0 2023-10-30T20:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.286+0000 7fa77532b640 1 monclient: found mon.noname-c 2023-10-30T20:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.286+0000 7fa77532b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:05.310 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.286+0000 7fa77532b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:24:05.310 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.286+0000 7fa77532b640 1 -- 172.21.15.69:0/2563563010 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7700b6e40 con 0x7fa7700b4390 2023-10-30T20:24:05.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.288+0000 7fa77532b640 1 --2- 172.21.15.69:0/2563563010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7700b4390 0x7fa7700b6730 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00d620 tx=0x7fa76c00dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:05.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.288+0000 7fa7667fc640 10 client.?.objecter ms_handle_connect 0x7fa7700b4390 2023-10-30T20:24:05.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.288+0000 7fa7667fc640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:05.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa7667fc640 1 -- 172.21.15.69:0/2563563010 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c00b620 con 0x7fa7700b4390 2023-10-30T20:24:05.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa7667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:05.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa7667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa7667fc640 10 monclient: dump: 2023-10-30T20:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa7667fc640 10 monclient: _finish_auth 0 2023-10-30T20:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa7667fc640 10 monclient: _check_auth_tickets 2023-10-30T20:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa7667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa7667fc640 1 -- 172.21.15.69:0/2563563010 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76c00b790 con 0x7fa7700b4390 2023-10-30T20:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa7667fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa7667fc640 1 -- 172.21.15.69:0/2563563010 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c00bb50 con 0x7fa7700b4390 2023-10-30T20:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa7667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa7667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa7667fc640 10 monclient: dump: 2023-10-30T20:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa777d90640 5 monclient: authenticate success, global_id 4244 2023-10-30T20:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa777d90640 10 monclient: _renew_subs 2023-10-30T20:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa777d90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:05.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa777d90640 1 -- 172.21.15.69:0/2563563010 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77013f840 con 0x7fa7700b4390 2023-10-30T20:24:05.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa767fff640 4 set_mon_vals no callback set 2023-10-30T20:24:05.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa777d90640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:05.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa777d90640 10 monclient: _renew_subs 2023-10-30T20:24:05.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa777d90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:05.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa777d90640 1 -- 172.21.15.69:0/2563563010 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7701daa50 con 0x7fa7700b4390 2023-10-30T20:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.292+0000 7fa777d90640 1 librados: init done 2023-10-30T20:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.293+0000 7fa777d90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.293+0000 7fa777d90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.293+0000 7fa777d90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.293+0000 7fa777d90640 1 -- 172.21.15.69:0/2563563010 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7700a9900 con 0x7fa7700b4390 2023-10-30T20:24:05.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.539+0000 7fa7667fc640 1 -- 172.21.15.69:0/2563563010 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fa76c00bcc0 con 0x7fa7700b4390 2023-10-30T20:24:05.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.540+0000 7fa7667fc640 1 --2- 172.21.15.69:0/2563563010 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa75406c810 0x7fa75406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:05.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.540+0000 7fa775b2c640 1 --2- 172.21.15.69:0/2563563010 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa75406c810 0x7fa75406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:05.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.540+0000 7fa775b2c640 10 monclient: get_auth_request con 0x7fa75406c810 auth_method 0 2023-10-30T20:24:05.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.540+0000 7fa7667fc640 10 client.4244.objecter ms_handle_connect 0x7fa75406c810 2023-10-30T20:24:05.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.540+0000 7fa775b2c640 1 --2- 172.21.15.69:0/2563563010 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa75406c810 0x7fa75406ecd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa760036000 tx=0x7fa760008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:05.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.554+0000 7fa7667fc640 1 -- 172.21.15.69:0/2563563010 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fa76c058e10 con 0x7fa7700b4390 2023-10-30T20:24:05.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.554+0000 7fa7667fc640 10 client.4244.objecter ms_dispatch 0x7fa770000b90 osd_map(21..21 src has 1..21) v4 2023-10-30T20:24:05.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.554+0000 7fa7667fc640 3 client.4244.objecter handle_osd_map got epochs [21,21] > 0 2023-10-30T20:24:05.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.554+0000 7fa7667fc640 3 client.4244.objecter handle_osd_map decoding full epoch 21 2023-10-30T20:24:05.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.554+0000 7fa7667fc640 20 client.4244.objecter dump_active .. 0 homeless 2023-10-30T20:24:05.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.554+0000 7fa7667fc640 1 -- 172.21.15.69:0/2563563010 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa76c019070 con 0x7fa7700b4390 2023-10-30T20:24:05.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.554+0000 7fa7667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:05.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.554+0000 7fa7667fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:05.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.744+0000 7fa777d90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:24:05.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.744+0000 7fa777d90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:24:05.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.744+0000 7fa777d90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:05.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:05.744+0000 7fa777d90640 1 -- 172.21.15.69:0/2563563010 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7701d9c70 con 0x7fa7700b4390 2023-10-30T20:24:06.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.053+0000 7fa7667fc640 1 -- 172.21.15.69:0/2563563010 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+12743 (secure 0 0 0) 0x7fa76c05ceb0 con 0x7fa7700b4390 2023-10-30T20:24:06.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.053+0000 7fa7667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:24:06.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.053+0000 7fa7667fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:06.056 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:24:06.056 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":21,"fsid":"32c1a723-7694-47d8-8b37-b1257dfe1796","created":"2023-10-30T20:23:06.559937+0000","modified":"2023-10-30T20:24:05.120527+0000","last_up_change":"2023-10-30T20:23:54.534733+0000","last_in_change":"2023-10-30T20:23:12.023084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T20:23:54.411443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T20:24:01.613740+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"af10bc15-318a-4349-af76-32a30c8acf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6801","nonce":2068724062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2068724062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":2068724062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2068724062}]},"public_addr":"172.21.15.69:6801/2068724062","cluster_addr":"172.21.15.69:6802/2068724062","heartbeat_back_addr":"172.21.15.69:6804/2068724062","heartbeat_front_addr":"172.21.15.69:6803/2068724062","state":["exists","up"]},{"osd":1,"uuid":"d0d32a9a-e24a-4d0c-9b90-125485d70672","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6813","nonce":431025540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":431025540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":431025540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":431025540}]},"public_addr":"172.21.15.69:6813/431025540","cluster_addr":"172.21.15.69:6814/431025540","heartbeat_back_addr":"172.21.15.69:6816/431025540","heartbeat_front_addr":"172.21.15.69:6815/431025540","state":["exists","up"]},{"osd":2,"uuid":"508f2c3a-9e54-46aa-b6c0-52fad97fd341","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6805","nonce":2300065874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2300065874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2300065874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2300065874}]},"public_addr":"172.21.15.69:6805/2300065874","cluster_addr":"172.21.15.69:6806/2300065874","heartbeat_back_addr":"172.21.15.69:6808/2300065874","heartbeat_front_addr":"172.21.15.69:6807/2300065874","state":["exists","up"]},{"osd":3,"uuid":"003a2613-1050-4a96-b927-0e5983687319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6809","nonce":1824616966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1824616966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1824616966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1824616966}]},"public_addr":"172.21.15.69:6809/1824616966","cluster_addr":"172.21.15.69:6810/1824616966","heartbeat_back_addr":"172.21.15.69:6812/1824616966","heartbeat_front_addr":"172.21.15.69:6811/1824616966","state":["exists","up"]},{"osd":4,"uuid":"678fc58e-ed6a-4fd0-9825-93297e3af6f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":4151811482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6801","nonce":4151811482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6803","nonce":4151811482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":4151811482}]},"public_addr":"172.21.15.183:6800/4151811482","cluster_addr":"172.21.15.183:6801/4151811482","heartbeat_back_addr":"172.21.15.183:6803/4151811482","heartbeat_front_addr":"172.21.15.183:6802/4151811482","state":["exists","up"]},{"osd":5,"uuid":"f945159a-a53c-48ba-97df-9374f9e56ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2276632263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6813","nonce":2276632263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6815","nonce":2276632263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2276632263}]},"public_addr":"172.21.15.183:6812/2276632263","cluster_addr":"172.21.15.183:6813/2276632263","heartbeat_back_addr":"172.21.15.183:6815/2276632263","heartbeat_front_addr":"172.21.15.183:6814/2276632263","state":["exists","up"]},{"osd":6,"uuid":"8184f59d-15e2-4bdc-90e3-fe1e231ca125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1046221745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":1046221745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6807","nonce":1046221745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1046221745}]},"public_addr":"172.21.15.183:6804/1046221745","cluster_addr":"172.21.15.183:6805/1046221745","heartbeat_back_addr":"172.21.15.183:6807/1046221745","heartbeat_front_addr":"172.21.15.183:6806/1046221745","state":["exists","up"]},{"osd":7,"uuid":"1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1361328034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1361328034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":1361328034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1361328034}]},"public_addr":"172.21.15.183:6808/1361328034","cluster_addr":"172.21.15.183:6809/1361328034","heartbeat_back_addr":"172.21.15.183:6811/1361328034","heartbeat_front_addr":"172.21.15.183:6810/1361328034","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.509883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.602550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:50.691692+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.518736+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:48.903386+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.213129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:50.677675+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:50.250250+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T20:24:06.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.055+0000 7fa777d90640 10 librados: watch_flush enter 2023-10-30T20:24:06.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.055+0000 7fa777d90640 10 librados: watch_flush exit 2023-10-30T20:24:06.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.055+0000 7fa777d90640 20 client.4244.objecter shutdown clearing up homeless session... 2023-10-30T20:24:06.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.055+0000 7fa777d90640 10 client.4244.objecter successfully canceled tick 2023-10-30T20:24:06.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.055+0000 7fa777d90640 1 -- 172.21.15.69:0/2563563010 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa75406c810 msgr2=0x7fa75406ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:06.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.055+0000 7fa777d90640 1 --2- 172.21.15.69:0/2563563010 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa75406c810 0x7fa75406ecd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa760036000 tx=0x7fa760008030 comp rx=0 tx=0).stop 2023-10-30T20:24:06.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.055+0000 7fa777d90640 10 monclient: shutdown 2023-10-30T20:24:06.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.055+0000 7fa777d90640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:06.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.055+0000 7fa777d90640 1 -- 172.21.15.69:0/2563563010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7700b4390 msgr2=0x7fa7700b6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:06.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.055+0000 7fa777d90640 1 --2- 172.21.15.69:0/2563563010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7700b4390 0x7fa7700b6730 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00d620 tx=0x7fa76c00dae0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.055+0000 7fa777d90640 1 -- 172.21.15.69:0/2563563010 shutdown_connections 2023-10-30T20:24:06.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.055+0000 7fa777d90640 1 --2- 172.21.15.69:0/2563563010 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa75406c810 0x7fa75406ecd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.055+0000 7fa777d90640 1 --2- 172.21.15.69:0/2563563010 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa7701418a0 0x7fa7701da330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.056+0000 7fa777d90640 1 --2- 172.21.15.69:0/2563563010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7700b4390 0x7fa7700b6730 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.056+0000 7fa777d90640 1 --2- 172.21.15.69:0/2563563010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7700b1b00 0x7fa7700b5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.056+0000 7fa777d90640 1 -- 172.21.15.69:0/2563563010 >> 172.21.15.69:0/2563563010 conn(0x7fa770065ea0 msgr2=0x7fa770139960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:06.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.056+0000 7fa777d90640 1 -- 172.21.15.69:0/2563563010 shutdown_connections 2023-10-30T20:24:06.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.056+0000 7fa777d90640 1 -- 172.21.15.69:0/2563563010 wait complete. 2023-10-30T20:24:06.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.056+0000 7fa777d90640 1 librados: shutdown 2023-10-30T20:24:06.069 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-30T20:24:06.069 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T20:24:06.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.170+0000 7fc0f79a9640 10 monclient: get_monmap_and_config 2023-10-30T20:24:06.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.170+0000 7fc0f79a9640 10 monclient: build_initial_monmap 2023-10-30T20:24:06.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.170+0000 7fc0f79a9640 10 monclient: monmap: 2023-10-30T20:24:06.172 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:06.172 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:06.172 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:06.171265+0000 2023-10-30T20:24:06.172 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:06.171265+0000 2023-10-30T20:24:06.173 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:06.173 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:06.173 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:06.173 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:06.173 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:06.174 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:06.174 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:06.174 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.170+0000 7fc0f79a9640 1 Processor -- start 2023-10-30T20:24:06.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 1 -- start start 2023-10-30T20:24:06.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 10 monclient: init 2023-10-30T20:24:06.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:06.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-10-30T20:24:06.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc0f00b4390 0x7fc0f00a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 10 monclient(hunting): picked mon.noname-c con 0x7fc0f00b4390 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0f01418a0 0x7fc0f0141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 10 monclient(hunting): picked mon.noname-a con 0x7fc0f01418a0 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc0f00b1b00 addr v1:172.21.15.69:6789/0 2023-10-30T20:24:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f00a8f60 con 0x7fc0f00b1b00 2023-10-30T20:24:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f0067a10 con 0x7fc0f01418a0 2023-10-30T20:24:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f00b4770 con 0x7fc0f00b4390 2023-10-30T20:24:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.171+0000 7fc0f79a9640 10 monclient(hunting): authenticate will time out at 1569.460611s 2023-10-30T20:24:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.172+0000 7fc0f4f44640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0f01418a0 0x7fc0f0141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:06.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.172+0000 7fc0f4f44640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0f01418a0 0x7fc0f0141ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45168/0 (socket says 172.21.15.69:45168) 2023-10-30T20:24:06.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.172+0000 7fc0f4f44640 1 -- 172.21.15.69:0/202993811 learned_addr learned my addr 172.21.15.69:0/202993811 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:06.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.172+0000 7fc0f4f44640 10 monclient(hunting): get_auth_request con 0x7fc0f01418a0 auth_method 0 2023-10-30T20:24:06.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.172+0000 7fc0f4f44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:06.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.172+0000 7fc0f4f44640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:06.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.172+0000 7fc0f4f44640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:06.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.173+0000 7fc0f4f44640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:06.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.173+0000 7fc0f4f44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:06.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.173+0000 7fc0f4f44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:06.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.174+0000 7fc0f5745640 1 --2- 172.21.15.69:0/202993811 >> v2:172.21.15.183:3300/0 conn(0x7fc0f00b4390 0x7fc0f00a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:06.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.174+0000 7fc0f5745640 10 monclient(hunting): get_auth_request con 0x7fc0f00b4390 auth_method 0 2023-10-30T20:24:06.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.174+0000 7fc0f5745640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:06.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.174+0000 7fc0f5745640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:06.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.174+0000 7fc0f5745640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:06.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.174+0000 7fc0f5745640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:06.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.174+0000 7fc0f5745640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:06.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.174+0000 7fc0f5745640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:06.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.176+0000 7fc0f5745640 10 monclient(hunting): handle_auth_done global_id 4258 payload 274 2023-10-30T20:24:06.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.176+0000 7fc0f5745640 1 -- 172.21.15.69:0/202993811 >> v2:172.21.15.69:3300/0 conn(0x7fc0f01418a0 msgr2=0x7fc0f0141ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:06.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.176+0000 7fc0f5745640 1 --2- 172.21.15.69:0/202993811 >> v2:172.21.15.69:3300/0 conn(0x7fc0f01418a0 0x7fc0f0141ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.176+0000 7fc0f5745640 1 -- 172.21.15.69:0/202993811 >> v1:172.21.15.69:6789/0 conn(0x7fc0f00b1b00 legacy=0x7fc0f00b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:06.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.176+0000 7fc0f5745640 10 monclient: _finish_hunting 0 2023-10-30T20:24:06.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.176+0000 7fc0f5745640 1 monclient: found mon.noname-c 2023-10-30T20:24:06.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.176+0000 7fc0f5745640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:06.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.176+0000 7fc0f5745640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:24:06.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.176+0000 7fc0f5745640 1 -- 172.21.15.69:0/202993811 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f00aa3d0 con 0x7fc0f00b4390 2023-10-30T20:24:06.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.178+0000 7fc0f5745640 1 --2- 172.21.15.69:0/202993811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0f00b4390 0x7fc0f00a8770 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0f00b4a50 tx=0x7fc0ec02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=494594f94959ccbb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:06.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0e7fff640 1 -- 172.21.15.69:0/202993811 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec031830 con 0x7fc0f00b4390 2023-10-30T20:24:06.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:06.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:06.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0e7fff640 10 monclient: dump: 2023-10-30T20:24:06.187 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:06.188 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:06.188 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:06.188 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:06.188 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:06.188 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:06.189 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:06.189 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:06.189 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:06.189 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0e7fff640 10 monclient: _finish_auth 0 2023-10-30T20:24:06.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0e7fff640 10 monclient: _check_auth_tickets 2023-10-30T20:24:06.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:06.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0e7fff640 1 -- 172.21.15.69:0/202993811 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ec02d040 con 0x7fc0f00b4390 2023-10-30T20:24:06.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0e7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:06.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0e7fff640 1 -- 172.21.15.69:0/202993811 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec031bc0 con 0x7fc0f00b4390 2023-10-30T20:24:06.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0f79a9640 5 monclient: authenticate success, global_id 4258 2023-10-30T20:24:06.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0f79a9640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:06.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0f79a9640 4 set_mon_vals no callback set 2023-10-30T20:24:06.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:06.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:06.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.182+0000 7fc0e7fff640 10 monclient: dump: 2023-10-30T20:24:06.191 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:06.193 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:06.193 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:06.193 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:06.193 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.183+0000 7fc0f79a9640 10 monclient: shutdown 2023-10-30T20:24:06.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.183+0000 7fc0f79a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:06.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.183+0000 7fc0f79a9640 1 -- 172.21.15.69:0/202993811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0f00b4390 msgr2=0x7fc0f00a8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:06.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.183+0000 7fc0f79a9640 1 --2- 172.21.15.69:0/202993811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0f00b4390 0x7fc0f00a8770 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0f00b4a50 tx=0x7fc0ec02b8d0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.183+0000 7fc0f79a9640 1 -- 172.21.15.69:0/202993811 shutdown_connections 2023-10-30T20:24:06.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.183+0000 7fc0f79a9640 1 --2- 172.21.15.69:0/202993811 >> v2:172.21.15.69:3300/0 conn(0x7fc0f01418a0 0x7fc0f0141ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.183+0000 7fc0f79a9640 1 --2- 172.21.15.69:0/202993811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0f00b4390 0x7fc0f00a8770 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.183+0000 7fc0f79a9640 1 -- 172.21.15.69:0/202993811 >> 172.21.15.69:0/202993811 conn(0x7fc0f0065ea0 msgr2=0x7fc0f00662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:06.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.183+0000 7fc0f79a9640 1 -- 172.21.15.69:0/202993811 shutdown_connections 2023-10-30T20:24:06.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.183+0000 7fc0f79a9640 1 -- 172.21.15.69:0/202993811 wait complete. 2023-10-30T20:24:06.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.184+0000 7fc0f79a9640 10 monclient: build_initial_monmap 2023-10-30T20:24:06.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.184+0000 7fc0f79a9640 10 monclient: monmap: 2023-10-30T20:24:06.199 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:06.199 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:06.200 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:06.200 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:06.200 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:06.200 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:06.200 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:06.200 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:06.201 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:06.201 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.185+0000 7fc0f79a9640 1 librados: starting msgr at 2023-10-30T20:24:06.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.185+0000 7fc0f79a9640 1 librados: starting objecter 2023-10-30T20:24:06.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.185+0000 7fc0f79a9640 1 Processor -- start 2023-10-30T20:24:06.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.185+0000 7fc0f79a9640 1 -- start start 2023-10-30T20:24:06.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.185+0000 7fc0f79a9640 1 librados: setting wanted keys 2023-10-30T20:24:06.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.185+0000 7fc0f79a9640 1 librados: calling monclient init 2023-10-30T20:24:06.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.185+0000 7fc0f79a9640 10 monclient: init 2023-10-30T20:24:06.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:06.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T20:24:06.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0f00b1b00 0x7fc0f01432d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 10 monclient(hunting): picked mon.noname-c con 0x7fc0f00b1b00 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:06.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0f00b4390 0x7fc0f0143ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 10 monclient(hunting): picked mon.noname-b con 0x7fc0f00b4390 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:06.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f01418a0 0x7fc0f01d6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 10 monclient(hunting): picked mon.noname-a con 0x7fc0f01418a0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:06.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f00b11d0 con 0x7fc0f01418a0 2023-10-30T20:24:06.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f0144400 con 0x7fc0f00b4390 2023-10-30T20:24:06.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f013f060 con 0x7fc0f00b1b00 2023-10-30T20:24:06.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:06.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f79a9640 10 monclient(hunting): authenticate will time out at 1569.475018s 2023-10-30T20:24:06.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f4f44640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0f00b4390 0x7fc0f0143ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:06.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.186+0000 7fc0f5745640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0f00b1b00 0x7fc0f01432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:06.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.187+0000 7fc0f4f44640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0f00b4390 0x7fc0f0143ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36392/0 (socket says 172.21.15.69:36392) 2023-10-30T20:24:06.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.187+0000 7fc0f4f44640 1 -- 172.21.15.69:0/3120384016 learned_addr learned my addr 172.21.15.69:0/3120384016 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:06.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.187+0000 7fc0f4f44640 10 monclient(hunting): get_auth_request con 0x7fc0f00b4390 auth_method 0 2023-10-30T20:24:06.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.187+0000 7fc0f4f44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:06.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.187+0000 7fc0f4f44640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:06.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.187+0000 7fc0f4f44640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:06.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.187+0000 7fc0f5745640 10 monclient(hunting): get_auth_request con 0x7fc0f00b1b00 auth_method 0 2023-10-30T20:24:06.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.187+0000 7fc0f5745640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:06.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.187+0000 7fc0f5745640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:06.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.187+0000 7fc0f5745640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:06.208 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.187+0000 7fc0f4f44640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:06.208 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.187+0000 7fc0f4f44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:06.208 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.187+0000 7fc0f4f44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:06.208 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.188+0000 7fc0f5745640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:06.208 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.188+0000 7fc0f5745640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:06.208 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.188+0000 7fc0f5745640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:06.209 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.190+0000 7fc0f4f44640 10 monclient(hunting): handle_auth_done global_id 4261 payload 824 2023-10-30T20:24:06.209 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.190+0000 7fc0f4f44640 1 -- 172.21.15.69:0/3120384016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0f00b1b00 msgr2=0x7fc0f01432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:06.209 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.190+0000 7fc0f4f44640 1 --2- 172.21.15.69:0/3120384016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0f00b1b00 0x7fc0f01432d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.209 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.190+0000 7fc0f4f44640 1 -- 172.21.15.69:0/3120384016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f01418a0 msgr2=0x7fc0f01d6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:06.209 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.190+0000 7fc0f4f44640 1 --2- 172.21.15.69:0/3120384016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f01418a0 0x7fc0f01d6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.210 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.190+0000 7fc0f4f44640 10 monclient: _finish_hunting 0 2023-10-30T20:24:06.210 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.190+0000 7fc0f4f44640 1 monclient: found mon.noname-b 2023-10-30T20:24:06.210 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.190+0000 7fc0f4f44640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:06.210 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.190+0000 7fc0f4f44640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:24:06.210 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.190+0000 7fc0f4f44640 1 -- 172.21.15.69:0/3120384016 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f013d190 con 0x7fc0f00b4390 2023-10-30T20:24:06.211 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.192+0000 7fc0f4f44640 1 --2- 172.21.15.69:0/3120384016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0f00b4390 0x7fc0f0143ec0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000e870 tx=0x7fc0e000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:06.211 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.192+0000 7fc0e5ffb640 10 client.?.objecter ms_handle_connect 0x7fc0f00b4390 2023-10-30T20:24:06.211 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.192+0000 7fc0e5ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:06.211 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0e5ffb640 1 -- 172.21.15.69:0/3120384016 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e000c8e0 con 0x7fc0f00b4390 2023-10-30T20:24:06.211 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:06.211 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0e5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:06.212 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0e5ffb640 10 monclient: dump: 2023-10-30T20:24:06.212 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:06.212 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:06.212 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:06.212 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:06.213 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:06.213 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:06.213 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:06.213 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:06.213 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:06.214 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.214 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0e5ffb640 10 monclient: _finish_auth 0 2023-10-30T20:24:06.214 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0e5ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:24:06.214 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:06.214 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0e5ffb640 1 -- 172.21.15.69:0/3120384016 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e000d040 con 0x7fc0f00b4390 2023-10-30T20:24:06.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0f79a9640 5 monclient: authenticate success, global_id 4261 2023-10-30T20:24:06.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0e5ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:06.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0f79a9640 10 monclient: _renew_subs 2023-10-30T20:24:06.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0e5ffb640 1 -- 172.21.15.69:0/3120384016 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e000cc90 con 0x7fc0f00b4390 2023-10-30T20:24:06.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0f79a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:06.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0f79a9640 1 -- 172.21.15.69:0/3120384016 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f01d7350 con 0x7fc0f00b4390 2023-10-30T20:24:06.216 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:06.216 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0e77fe640 4 set_mon_vals no callback set 2023-10-30T20:24:06.216 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:06.216 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0e5ffb640 10 monclient: dump: 2023-10-30T20:24:06.216 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:06.216 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:06.217 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:06.217 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:06.217 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:06.217 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:06.217 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:06.218 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:06.218 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:06.218 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0f79a9640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:06.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0f79a9640 10 monclient: _renew_subs 2023-10-30T20:24:06.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0f79a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:06.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0f79a9640 1 -- 172.21.15.69:0/3120384016 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f0135370 con 0x7fc0f00b4390 2023-10-30T20:24:06.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.196+0000 7fc0f79a9640 1 librados: init done 2023-10-30T20:24:06.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.197+0000 7fc0f79a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:06.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.197+0000 7fc0f79a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:06.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.197+0000 7fc0f79a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:06.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.197+0000 7fc0f79a9640 1 -- 172.21.15.69:0/3120384016 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f01d6c60 con 0x7fc0f00b4390 2023-10-30T20:24:06.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.215+0000 7fc0e5ffb640 1 -- 172.21.15.69:0/3120384016 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fc0e0013470 con 0x7fc0f00b4390 2023-10-30T20:24:06.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.216+0000 7fc0e5ffb640 1 --2- 172.21.15.69:0/3120384016 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc0d406c7c0 0x7fc0d406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.216+0000 7fc0f5745640 1 --2- 172.21.15.69:0/3120384016 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc0d406c7c0 0x7fc0d406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:06.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.216+0000 7fc0f5745640 10 monclient: get_auth_request con 0x7fc0d406c7c0 auth_method 0 2023-10-30T20:24:06.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.216+0000 7fc0e5ffb640 10 client.4261.objecter ms_handle_connect 0x7fc0d406c7c0 2023-10-30T20:24:06.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.216+0000 7fc0f5745640 1 --2- 172.21.15.69:0/3120384016 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc0d406c7c0 0x7fc0d406ec80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec036000 tx=0x7fc0ec03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:06.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.445+0000 7fc0e5ffb640 1 -- 172.21.15.69:0/3120384016 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fc0e00586e0 con 0x7fc0f00b4390 2023-10-30T20:24:06.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.445+0000 7fc0e5ffb640 10 client.4261.objecter ms_dispatch 0x7fc0f0000b90 osd_map(21..21 src has 1..21) v4 2023-10-30T20:24:06.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.445+0000 7fc0e5ffb640 3 client.4261.objecter handle_osd_map got epochs [21,21] > 0 2023-10-30T20:24:06.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.445+0000 7fc0e5ffb640 3 client.4261.objecter handle_osd_map decoding full epoch 21 2023-10-30T20:24:06.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.445+0000 7fc0e5ffb640 20 client.4261.objecter dump_active .. 0 homeless 2023-10-30T20:24:06.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.456+0000 7fc0e5ffb640 1 -- 172.21.15.69:0/3120384016 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc0e005c780 con 0x7fc0f00b4390 2023-10-30T20:24:06.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.456+0000 7fc0e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:06.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.456+0000 7fc0e5ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:06.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.641+0000 7fc0f79a9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:24:06.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.641+0000 7fc0f79a9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:24:06.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.641+0000 7fc0f79a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:06.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.641+0000 7fc0f79a9640 1 -- 172.21.15.69:0/3120384016 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0f00a9900 con 0x7fc0f00b4390 2023-10-30T20:24:06.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.716+0000 7fc0e5ffb640 1 -- 172.21.15.69:0/3120384016 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+12743 (secure 0 0 0) 0x7fc0e0060930 con 0x7fc0f00b4390 2023-10-30T20:24:06.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.716+0000 7fc0e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:24:06.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.716+0000 7fc0e5ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:06.719 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:24:06.719 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":21,"fsid":"32c1a723-7694-47d8-8b37-b1257dfe1796","created":"2023-10-30T20:23:06.559937+0000","modified":"2023-10-30T20:24:05.120527+0000","last_up_change":"2023-10-30T20:23:54.534733+0000","last_in_change":"2023-10-30T20:23:12.023084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T20:23:54.411443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T20:24:01.613740+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"af10bc15-318a-4349-af76-32a30c8acf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6801","nonce":2068724062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2068724062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":2068724062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2068724062}]},"public_addr":"172.21.15.69:6801/2068724062","cluster_addr":"172.21.15.69:6802/2068724062","heartbeat_back_addr":"172.21.15.69:6804/2068724062","heartbeat_front_addr":"172.21.15.69:6803/2068724062","state":["exists","up"]},{"osd":1,"uuid":"d0d32a9a-e24a-4d0c-9b90-125485d70672","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6813","nonce":431025540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":431025540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":431025540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":431025540}]},"public_addr":"172.21.15.69:6813/431025540","cluster_addr":"172.21.15.69:6814/431025540","heartbeat_back_addr":"172.21.15.69:6816/431025540","heartbeat_front_addr":"172.21.15.69:6815/431025540","state":["exists","up"]},{"osd":2,"uuid":"508f2c3a-9e54-46aa-b6c0-52fad97fd341","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6805","nonce":2300065874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2300065874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2300065874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2300065874}]},"public_addr":"172.21.15.69:6805/2300065874","cluster_addr":"172.21.15.69:6806/2300065874","heartbeat_back_addr":"172.21.15.69:6808/2300065874","heartbeat_front_addr":"172.21.15.69:6807/2300065874","state":["exists","up"]},{"osd":3,"uuid":"003a2613-1050-4a96-b927-0e5983687319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6809","nonce":1824616966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1824616966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1824616966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1824616966}]},"public_addr":"172.21.15.69:6809/1824616966","cluster_addr":"172.21.15.69:6810/1824616966","heartbeat_back_addr":"172.21.15.69:6812/1824616966","heartbeat_front_addr":"172.21.15.69:6811/1824616966","state":["exists","up"]},{"osd":4,"uuid":"678fc58e-ed6a-4fd0-9825-93297e3af6f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":4151811482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6801","nonce":4151811482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6803","nonce":4151811482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":4151811482}]},"public_addr":"172.21.15.183:6800/4151811482","cluster_addr":"172.21.15.183:6801/4151811482","heartbeat_back_addr":"172.21.15.183:6803/4151811482","heartbeat_front_addr":"172.21.15.183:6802/4151811482","state":["exists","up"]},{"osd":5,"uuid":"f945159a-a53c-48ba-97df-9374f9e56ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2276632263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6813","nonce":2276632263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6815","nonce":2276632263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2276632263}]},"public_addr":"172.21.15.183:6812/2276632263","cluster_addr":"172.21.15.183:6813/2276632263","heartbeat_back_addr":"172.21.15.183:6815/2276632263","heartbeat_front_addr":"172.21.15.183:6814/2276632263","state":["exists","up"]},{"osd":6,"uuid":"8184f59d-15e2-4bdc-90e3-fe1e231ca125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1046221745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":1046221745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6807","nonce":1046221745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1046221745}]},"public_addr":"172.21.15.183:6804/1046221745","cluster_addr":"172.21.15.183:6805/1046221745","heartbeat_back_addr":"172.21.15.183:6807/1046221745","heartbeat_front_addr":"172.21.15.183:6806/1046221745","state":["exists","up"]},{"osd":7,"uuid":"1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1361328034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1361328034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":1361328034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1361328034}]},"public_addr":"172.21.15.183:6808/1361328034","cluster_addr":"172.21.15.183:6809/1361328034","heartbeat_back_addr":"172.21.15.183:6811/1361328034","heartbeat_front_addr":"172.21.15.183:6810/1361328034","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.509883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.602550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:50.691692+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.518736+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:48.903386+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.213129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:50.677675+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:50.250250+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T20:24:06.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 10 librados: watch_flush enter 2023-10-30T20:24:06.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 10 librados: watch_flush exit 2023-10-30T20:24:06.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 20 client.4261.objecter shutdown clearing up homeless session... 2023-10-30T20:24:06.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 10 client.4261.objecter successfully canceled tick 2023-10-30T20:24:06.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 1 -- 172.21.15.69:0/3120384016 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc0d406c7c0 msgr2=0x7fc0d406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:06.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 1 --2- 172.21.15.69:0/3120384016 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc0d406c7c0 0x7fc0d406ec80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec036000 tx=0x7fc0ec03c040 comp rx=0 tx=0).stop 2023-10-30T20:24:06.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 10 monclient: shutdown 2023-10-30T20:24:06.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:06.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 1 -- 172.21.15.69:0/3120384016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0f00b4390 msgr2=0x7fc0f0143ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:06.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 1 --2- 172.21.15.69:0/3120384016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0f00b4390 0x7fc0f0143ec0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000e870 tx=0x7fc0e000ed30 comp rx=0 tx=0).stop 2023-10-30T20:24:06.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 1 -- 172.21.15.69:0/3120384016 shutdown_connections 2023-10-30T20:24:06.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 1 --2- 172.21.15.69:0/3120384016 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc0d406c7c0 0x7fc0d406ec80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 1 --2- 172.21.15.69:0/3120384016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f01418a0 0x7fc0f01d6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 1 --2- 172.21.15.69:0/3120384016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0f00b4390 0x7fc0f0143ec0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 1 --2- 172.21.15.69:0/3120384016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0f00b1b00 0x7fc0f01432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 1 -- 172.21.15.69:0/3120384016 >> 172.21.15.69:0/3120384016 conn(0x7fc0f0065ea0 msgr2=0x7fc0f00737b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 1 -- 172.21.15.69:0/3120384016 shutdown_connections 2023-10-30T20:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 1 -- 172.21.15.69:0/3120384016 wait complete. 2023-10-30T20:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.719+0000 7fc0f79a9640 1 librados: shutdown 2023-10-30T20:24:06.731 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-30T20:24:06.732 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-30T20:24:06.733 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-30T20:24:06.733 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-30T20:24:06.734 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-30T20:24:06.734 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-30T20:24:06.735 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-30T20:24:06.735 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-30T20:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.910+0000 7f1cf984b640 10 monclient: get_monmap_and_config 2023-10-30T20:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.910+0000 7f1cf984b640 10 monclient: build_initial_monmap 2023-10-30T20:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.910+0000 7f1cf984b640 10 monclient: monmap: 2023-10-30T20:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:06.911491+0000 2023-10-30T20:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:06.911491+0000 2023-10-30T20:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:06.914 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:06.914 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:06.914 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:06.914 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:06.914 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:06.915 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.911+0000 7f1cf984b640 1 Processor -- start 2023-10-30T20:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.911+0000 7f1cf984b640 1 -- start start 2023-10-30T20:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.911+0000 7f1cf984b640 10 monclient: init 2023-10-30T20:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.912+0000 7f1cf984b640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.912+0000 7f1cf984b640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-10-30T20:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.912+0000 7f1cf984b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1cec0070e0 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.912+0000 7f1cf984b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1cec036530 0x7f1cec033600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.912+0000 7f1cf984b640 10 monclient(hunting): picked mon.noname-c con 0x7f1cec036530 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.912+0000 7f1cf984b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1cec030b10 addr v1:172.21.15.69:6789/0 2023-10-30T20:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.912+0000 7f1cf984b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec0355f0 con 0x7f1cec030b10 2023-10-30T20:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.912+0000 7f1cf984b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec037580 con 0x7f1cec0070e0 2023-10-30T20:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.912+0000 7f1cf984b640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.912+0000 7f1cf984b640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cec008600 con 0x7f1cec036530 2023-10-30T20:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.912+0000 7f1cf984b640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.912+0000 7f1cf984b640 10 monclient(hunting): authenticate will time out at 1570.201409s 2023-10-30T20:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.913+0000 7f1cf27fc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1cec036530 0x7f1cec033600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:06.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.913+0000 7f0f7fa2d640 10 monclient: get_monmap_and_config 2023-10-30T20:24:06.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.913+0000 7f0f7fa2d640 10 monclient: build_initial_monmap 2023-10-30T20:24:06.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.914+0000 7f0f7fa2d640 10 monclient: monmap: 2023-10-30T20:24:06.921 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:06.921 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:06.921 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:06.914973+0000 2023-10-30T20:24:06.921 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:06.914973+0000 2023-10-30T20:24:06.921 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:06.921 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:06.922 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:06.922 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:06.922 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:06.922 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:06.923 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:06.923 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.914+0000 7f1cf27fc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1cec036530 0x7f1cec033600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36402/0 (socket says 172.21.15.69:36402) 2023-10-30T20:24:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.914+0000 7f1cf27fc640 1 -- 172.21.15.69:0/3775449672 learned_addr learned my addr 172.21.15.69:0/3775449672 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.914+0000 7f1cf27fc640 10 monclient(hunting): get_auth_request con 0x7f1cec036530 auth_method 0 2023-10-30T20:24:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.914+0000 7f1cf27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.914+0000 7f1cf27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.914+0000 7f1cf27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.915+0000 7f1cf27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.915+0000 7f1cf27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:06.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.915+0000 7f1cf27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:06.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.914+0000 7f0f7fa2d640 1 Processor -- start 2023-10-30T20:24:06.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.915+0000 7f0f7fa2d640 1 -- start start 2023-10-30T20:24:06.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.915+0000 7f0f7fa2d640 10 monclient: init 2023-10-30T20:24:06.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.916+0000 7f0f7fa2d640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:06.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.916+0000 7f0f7fa2d640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-10-30T20:24:06.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.916+0000 7f0f7fa2d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f700070e0 0x7f0f70035e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.916+0000 7f0f7fa2d640 10 monclient(hunting): picked mon.noname-b con 0x7f0f700070e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:06.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.916+0000 7f0f7fa2d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0f70030b10 0x7f0f70036360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.916+0000 7f0f7fa2d640 10 monclient(hunting): picked mon.noname-c con 0x7f0f70030b10 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:06.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.916+0000 7f0f7fa2d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f70033600 addr v1:172.21.15.69:6789/0 2023-10-30T20:24:06.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.916+0000 7f0f7fa2d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f700355f0 con 0x7f0f70033600 2023-10-30T20:24:06.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.916+0000 7f0f7fa2d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.916+0000 7f0f7fa2d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f70008600 con 0x7f0f70030b10 2023-10-30T20:24:06.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.916+0000 7f0f7fa2d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.916+0000 7f0f7fa2d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f700074c0 con 0x7f0f700070e0 2023-10-30T20:24:06.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.916+0000 7f0f7fa2d640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:06.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.916+0000 7f0f7fa2d640 10 monclient(hunting): authenticate will time out at 1570.204951s 2023-10-30T20:24:06.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.918+0000 7f0f7d7c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f700070e0 0x7f0f70035e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:06.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.918+0000 7f0f7d7c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f700070e0 0x7f0f70035e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60622/0 (socket says 172.21.15.69:60622) 2023-10-30T20:24:06.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.918+0000 7f0f7d7c9640 1 -- 172.21.15.69:0/2239168797 learned_addr learned my addr 172.21.15.69:0/2239168797 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:06.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.918+0000 7f0f7d7c9640 10 monclient(hunting): get_auth_request con 0x7f0f700070e0 auth_method 0 2023-10-30T20:24:06.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.918+0000 7f0f7d7c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:06.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.918+0000 7f0f7d7c9640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:06.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.918+0000 7f0f7d7c9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:06.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.919+0000 7f0f7d7c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:06.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.919+0000 7f0f7d7c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:06.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.919+0000 7f0f7d7c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:06.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.922+0000 7f0f7cfc8640 1 --2- 172.21.15.69:0/2239168797 >> v2:172.21.15.183:3300/0 conn(0x7f0f70030b10 0x7f0f70036360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:06.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.922+0000 7f0f7cfc8640 10 monclient(hunting): get_auth_request con 0x7f0f70030b10 auth_method 0 2023-10-30T20:24:06.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.922+0000 7f0f7cfc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:06.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.922+0000 7f0f7cfc8640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:06.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.922+0000 7f0f7cfc8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:06.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.922+0000 7f9c00541640 10 monclient: get_monmap_and_config 2023-10-30T20:24:06.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.922+0000 7f9c00541640 10 monclient: build_initial_monmap 2023-10-30T20:24:06.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.922+0000 7f9c00541640 10 monclient: monmap: 2023-10-30T20:24:06.937 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:06.937 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:06.937 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:06.923827+0000 2023-10-30T20:24:06.938 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:06.923827+0000 2023-10-30T20:24:06.938 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:06.938 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:06.938 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:06.939 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:06.939 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:06.939 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:06.939 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:06.940 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.923+0000 7f9c00541640 1 Processor -- start 2023-10-30T20:24:06.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.923+0000 7f9c00541640 1 -- start start 2023-10-30T20:24:06.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.923+0000 7f9c00541640 10 monclient: init 2023-10-30T20:24:06.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.924+0000 7f9c00541640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:06.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.924+0000 7f9c00541640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-10-30T20:24:06.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.921+0000 7f1cf27fc640 10 monclient(hunting): handle_auth_done global_id 4264 payload 274 2023-10-30T20:24:06.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.921+0000 7f1cf27fc640 1 -- 172.21.15.69:0/3775449672 >> v1:172.21.15.183:6789/0 conn(0x7f1cec0070e0 legacy=0x7f1cec035e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:06.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.921+0000 7f1cf27fc640 1 -- 172.21.15.69:0/3775449672 >> v1:172.21.15.69:6789/0 conn(0x7f1cec030b10 legacy=0x7f1cec0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:06.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.921+0000 7f1cf27fc640 10 monclient: _finish_hunting 0 2023-10-30T20:24:06.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.921+0000 7f1cf27fc640 1 monclient: found mon.noname-c 2023-10-30T20:24:06.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.921+0000 7f1cf27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:06.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.921+0000 7f1cf27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:24:06.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.921+0000 7f1cf27fc640 1 -- 172.21.15.69:0/3775449672 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec035890 con 0x7f1cec036530 2023-10-30T20:24:06.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.924+0000 7f9c00541640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9bf00070e0 0x7f9bf0035e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.924+0000 7f9c00541640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf00070e0 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:06.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.925+0000 7f9c00541640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9bf0030b10 addr v1:172.21.15.69:6789/0 2023-10-30T20:24:06.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.925+0000 7f9c00541640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9bf0033d10 0x7f9bf0036360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.925+0000 7f9c00541640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf0033d10 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:06.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.925+0000 7f9c00541640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf00355f0 con 0x7f9bf0030b10 2023-10-30T20:24:06.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.925+0000 7f9c00541640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.925+0000 7f9c00541640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf0008600 con 0x7f9bf0033d10 2023-10-30T20:24:06.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.925+0000 7f9c00541640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.925+0000 7f9c00541640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf00074c0 con 0x7f9bf00070e0 2023-10-30T20:24:06.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.925+0000 7f9c00541640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:06.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.925+0000 7f9c00541640 10 monclient(hunting): authenticate will time out at 1570.213865s 2023-10-30T20:24:06.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.923+0000 7f0f7cfc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:06.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.923+0000 7f0f7cfc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:06.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.923+0000 7f0f7cfc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:06.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.924+0000 7f0f7d7c9640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2023-10-30T20:24:06.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.924+0000 7f0f7d7c9640 1 -- 172.21.15.69:0/2239168797 >> v2:172.21.15.183:3300/0 conn(0x7f0f70030b10 msgr2=0x7f0f70036360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:06.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.924+0000 7f0f7d7c9640 1 --2- 172.21.15.69:0/2239168797 >> v2:172.21.15.183:3300/0 conn(0x7f0f70030b10 0x7f0f70036360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.924+0000 7f0f7d7c9640 1 -- 172.21.15.69:0/2239168797 >> v1:172.21.15.69:6789/0 conn(0x7f0f70033600 legacy=0x7f0f70033a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:06.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.924+0000 7f0f7d7c9640 10 monclient: _finish_hunting 0 2023-10-30T20:24:06.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.924+0000 7f0f7d7c9640 1 monclient: found mon.noname-b 2023-10-30T20:24:06.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.924+0000 7f0f7d7c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:06.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.924+0000 7f0f7d7c9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:24:06.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.924+0000 7f0f7d7c9640 1 -- 172.21.15.69:0/2239168797 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f70037580 con 0x7f0f700070e0 2023-10-30T20:24:06.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.927+0000 7f9bfe2dd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9bf00070e0 0x7f9bf0035e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:06.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.928+0000 7f1cf27fc640 1 --2- 172.21.15.69:0/3775449672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1cec036530 0x7f1cec033600 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1cf402cb40 tx=0x7f1cf40a5ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2427a74e3e13eeee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:06.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.928+0000 7f0f7d7c9640 1 --2- 172.21.15.69:0/2239168797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f700070e0 0x7f0f70035e20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009870 tx=0x7f0f7402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab40ce31aa41aba9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:06.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.928+0000 7f9bfdadc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9bf0030b10 0x7f9bf0033600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47850/0 (socket says 172.21.15.69:47850) 2023-10-30T20:24:06.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.928+0000 7f9bfdadc640 1 -- 172.21.15.69:0/2392942984 learned_addr learned my addr 172.21.15.69:0/2392942984 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:24:06.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.929+0000 7f9bfe2dd640 10 monclient(hunting): get_auth_request con 0x7f9bf00070e0 auth_method 0 2023-10-30T20:24:06.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.929+0000 7f9bfe2dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:06.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.929+0000 7f9bfe2dd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:06.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.929+0000 7f9bfe2dd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:06.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.929+0000 7f9bfeade640 1 --2- 172.21.15.69:0/2392942984 >> v2:172.21.15.69:3300/0 conn(0x7f9bf0033d10 0x7f9bf0036360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:06.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.929+0000 7f9bfeade640 10 monclient(hunting): get_auth_request con 0x7f9bf0033d10 auth_method 0 2023-10-30T20:24:06.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.929+0000 7f9bfeade640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:06.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.929+0000 7f9bfeade640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:06.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.929+0000 7f9bfeade640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:06.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.930+0000 7f9bfe2dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:06.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.930+0000 7f9bfe2dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:06.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.930+0000 7f9bfe2dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:06.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.929+0000 7fb993ade640 10 monclient: get_monmap_and_config 2023-10-30T20:24:06.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.929+0000 7fb993ade640 10 monclient: build_initial_monmap 2023-10-30T20:24:06.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.929+0000 7fb993ade640 10 monclient: monmap: 2023-10-30T20:24:06.956 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:06.930675+0000 2023-10-30T20:24:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:06.930675+0000 2023-10-30T20:24:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:06.958 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:06.958 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:06.958 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:06.958 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:06.959 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:06.959 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.930+0000 7f9bfeade640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:06.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.930+0000 7f9bfeade640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:06.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.930+0000 7f9bfeade640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:06.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.930+0000 7fb993ade640 1 Processor -- start 2023-10-30T20:24:06.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.931+0000 7fb993ade640 1 -- start start 2023-10-30T20:24:06.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.931+0000 7fb993ade640 10 monclient: init 2023-10-30T20:24:06.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.931+0000 7f462a7f5640 10 monclient: get_monmap_and_config 2023-10-30T20:24:06.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.931+0000 7f462a7f5640 10 monclient: build_initial_monmap 2023-10-30T20:24:06.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.931+0000 7f462a7f5640 10 monclient: monmap: 2023-10-30T20:24:06.962 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:06.962 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:06.962 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:06.932051+0000 2023-10-30T20:24:06.962 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:06.932051+0000 2023-10-30T20:24:06.963 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:06.963 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:06.963 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:06.963 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:06.964 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:06.964 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:06.964 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:06.964 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7f462a7f5640 1 Processor -- start 2023-10-30T20:24:06.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.931+0000 7fb993ade640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:06.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.931+0000 7fb993ade640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-10-30T20:24:06.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.931+0000 7fb993ade640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb98c0aca50 0x7fb98c06db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.931+0000 7fb993ade640 10 monclient(hunting): picked mon.noname-c con 0x7fb98c0aca50 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.931+0000 7fb993ade640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb98c13eb80 0x7fb98c13efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.931+0000 7fb993ade640 10 monclient(hunting): picked mon.noname-b con 0x7fb98c13eb80 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.931+0000 7fb993ade640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb98c145360 0x7fb98c1457f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7fb993ade640 10 monclient(hunting): picked mon.noname-a con 0x7fb98c145360 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:06.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7fb993ade640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7fb993ade640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb98c067b90 con 0x7fb98c145360 2023-10-30T20:24:06.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7fb993ade640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7fb993ade640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb98c0ace30 con 0x7fb98c0aca50 2023-10-30T20:24:06.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7fb993ade640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7fb993ade640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c145d30 con 0x7fb98c13eb80 2023-10-30T20:24:06.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7fb993ade640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:06.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7fb993ade640 10 monclient(hunting): authenticate will time out at 1570.221051s 2023-10-30T20:24:06.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7f462a7f5640 1 -- start start 2023-10-30T20:24:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7f462a7f5640 10 monclient: init 2023-10-30T20:24:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.933+0000 7f462a7f5640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.933+0000 7f462a7f5640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-10-30T20:24:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.933+0000 7f462a7f5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4624071710 0x7f4624145df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.933+0000 7f462a7f5640 10 monclient(hunting): picked mon.noname-b con 0x7f4624071710 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.933+0000 7f462a7f5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4624071af0 0x7f46240ae690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.933+0000 7f462a7f5640 10 monclient(hunting): picked mon.noname-a con 0x7f4624071af0 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:06.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.933+0000 7f462a7f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4624143350 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:06.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.933+0000 7f462a7f5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f462406fec0 con 0x7f4624143350 2023-10-30T20:24:06.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.933+0000 7f462a7f5640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.933+0000 7f462a7f5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4624067b90 con 0x7f4624071af0 2023-10-30T20:24:06.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.933+0000 7f462a7f5640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:06.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.933+0000 7f462a7f5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4624071260 con 0x7f4624071710 2023-10-30T20:24:06.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.933+0000 7f462a7f5640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:06.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.933+0000 7f462a7f5640 10 monclient(hunting): authenticate will time out at 1570.222081s 2023-10-30T20:24:06.972 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7f9bfe2dd640 10 monclient(hunting): handle_auth_done global_id 4273 payload 274 2023-10-30T20:24:06.972 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7f9bfe2dd640 1 -- 172.21.15.69:0/2392942984 >> v2:172.21.15.69:3300/0 conn(0x7f9bf0033d10 msgr2=0x7f9bf0036360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:06.972 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7f9bfe2dd640 1 --2- 172.21.15.69:0/2392942984 >> v2:172.21.15.69:3300/0 conn(0x7f9bf0033d10 0x7f9bf0036360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.973 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7f9bfe2dd640 1 -- 172.21.15.69:0/2392942984 >> v1:172.21.15.69:6789/0 conn(0x7f9bf0030b10 legacy=0x7f9bf0033600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:06.973 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7f9bfe2dd640 10 monclient: _finish_hunting 0 2023-10-30T20:24:06.973 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7f9bfe2dd640 1 monclient: found mon.noname-c 2023-10-30T20:24:06.973 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7f9bfe2dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:06.973 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7f9bfe2dd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:24:06.973 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.932+0000 7f9bfe2dd640 1 -- 172.21.15.69:0/2392942984 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf0037580 con 0x7f9bf00070e0 2023-10-30T20:24:06.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f6ffff640 1 -- 172.21.15.69:0/2239168797 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f74030840 con 0x7f0f700070e0 2023-10-30T20:24:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f6ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f6ffff640 10 monclient: dump: 2023-10-30T20:24:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:06.977 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:06.977 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:06.977 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:06.977 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:06.977 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:06.978 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:06.978 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:06.978 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f6ffff640 10 monclient: _finish_auth 0 2023-10-30T20:24:06.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f6ffff640 10 monclient: _check_auth_tickets 2023-10-30T20:24:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f6ffff640 1 -- 172.21.15.69:0/2239168797 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f740309b0 con 0x7f0f700070e0 2023-10-30T20:24:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f6ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f6ffff640 1 -- 172.21.15.69:0/2239168797 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f74030d50 con 0x7f0f700070e0 2023-10-30T20:24:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f6ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:06.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f6ffff640 10 monclient: dump: 2023-10-30T20:24:06.980 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:06.980 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:06.980 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:06.980 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:06.981 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:06.981 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:06.981 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:06.981 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:06.981 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:06.982 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f7fa2d640 5 monclient: authenticate success, global_id 4253 2023-10-30T20:24:06.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f7fa2d640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:06.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.934+0000 7f0f7fa2d640 4 set_mon_vals no callback set 2023-10-30T20:24:06.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f4623fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4624071710 0x7f4624145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:06.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f1cf1ffb640 1 -- 172.21.15.69:0/3775449672 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf40a7930 con 0x7f1cec036530 2023-10-30T20:24:06.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f1cf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:06.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f1cf1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:06.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.936+0000 7f1cf1ffb640 10 monclient: dump: 2023-10-30T20:24:06.984 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:06.984 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:06.984 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:06.984 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:06.985 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:06.985 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:06.985 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:06.985 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:06.985 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:06.986 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.936+0000 7f1cf1ffb640 10 monclient: _finish_auth 0 2023-10-30T20:24:06.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.936+0000 7f1cf1ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:24:06.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.936+0000 7f1cf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:06.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.936+0000 7f1cf984b640 5 monclient: authenticate success, global_id 4264 2023-10-30T20:24:06.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.936+0000 7f1cf984b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:06.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.936+0000 7f1cf1ffb640 1 -- 172.21.15.69:0/3775449672 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf40a7aa0 con 0x7f1cec036530 2023-10-30T20:24:06.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.936+0000 7f1cf1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:06.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f4623fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4624071710 0x7f4624145df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60628/0 (socket says 172.21.15.69:60628) 2023-10-30T20:24:06.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f4623fff640 1 -- 172.21.15.69:0/307714617 learned_addr learned my addr 172.21.15.69:0/307714617 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:06.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f4623fff640 10 monclient(hunting): get_auth_request con 0x7f4624071710 auth_method 0 2023-10-30T20:24:06.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f4623fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:06.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f4623fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:06.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f4623fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:06.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.936+0000 7f1cf984b640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:06.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.936+0000 7f1cf984b640 4 set_mon_vals no callback set 2023-10-30T20:24:06.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f1cf984b640 10 monclient: shutdown 2023-10-30T20:24:06.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f1cf984b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:06.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f1cf984b640 1 -- 172.21.15.69:0/3775449672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1cec036530 msgr2=0x7f1cec033600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:06.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f1cf984b640 1 --2- 172.21.15.69:0/3775449672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1cec036530 0x7f1cec033600 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1cf402cb40 tx=0x7f1cf40a5ab0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f0f7fa2d640 10 monclient: shutdown 2023-10-30T20:24:06.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f0f7fa2d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:06.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2239168797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f700070e0 msgr2=0x7f0f70035e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f0f7fa2d640 1 --2- 172.21.15.69:0/2239168797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f700070e0 0x7f0f70035e20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009870 tx=0x7f0f7402b980 comp rx=0 tx=0).stop 2023-10-30T20:24:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2239168797 shutdown_connections 2023-10-30T20:24:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f0f7fa2d640 1 --2- 172.21.15.69:0/2239168797 >> v2:172.21.15.183:3300/0 conn(0x7f0f70030b10 0x7f0f70036360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f0f7fa2d640 1 --2- 172.21.15.69:0/2239168797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f700070e0 0x7f0f70035e20 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009870 tx=0x7f0f7402b980 comp rx=0 tx=0).stop 2023-10-30T20:24:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.935+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2239168797 >> 172.21.15.69:0/2239168797 conn(0x7f0f70006030 msgr2=0x7f0f70006440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:06.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.936+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2239168797 shutdown_connections 2023-10-30T20:24:06.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.936+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2239168797 wait complete. 2023-10-30T20:24:06.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.936+0000 7f0f7fa2d640 10 monclient: build_initial_monmap 2023-10-30T20:24:06.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.936+0000 7f0f7fa2d640 10 monclient: monmap: 2023-10-30T20:24:06.992 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:06.993 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:06.993 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:06.993 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:06.993 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:06.993 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:06.994 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:06.994 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:06.994 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:06.994 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:06.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f4623fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:06.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f4623fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:06.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f4623fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:06.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f1cf984b640 1 -- 172.21.15.69:0/3775449672 shutdown_connections 2023-10-30T20:24:06.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f1cf984b640 1 --2- 172.21.15.69:0/3775449672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1cec036530 0x7f1cec033600 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:06.996 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f1cf984b640 1 -- 172.21.15.69:0/3775449672 >> 172.21.15.69:0/3775449672 conn(0x7f1cec006030 msgr2=0x7f1cec006440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:06.996 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f0f7fa2d640 1 librados: starting msgr at 2023-10-30T20:24:06.996 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f0f7fa2d640 1 librados: starting objecter 2023-10-30T20:24:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f0f7fa2d640 1 Processor -- start 2023-10-30T20:24:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f0f7fa2d640 1 -- start start 2023-10-30T20:24:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f0f7fa2d640 1 librados: setting wanted keys 2023-10-30T20:24:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f0f7fa2d640 1 librados: calling monclient init 2023-10-30T20:24:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f0f7fa2d640 10 monclient: init 2023-10-30T20:24:06.998 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f1cf984b640 1 -- 172.21.15.69:0/3775449672 shutdown_connections 2023-10-30T20:24:06.999 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:06.999 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:24:06.999 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f70030b10 0x7f0f700c6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:06.999 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 10 monclient(hunting): picked mon.noname-a con 0x7f0f70030b10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f70033600 0x7f0f700c7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 10 monclient(hunting): picked mon.noname-b con 0x7f0f70033600 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f7015d340 0x7f0f7015d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 10 monclient(hunting): picked mon.noname-c con 0x7f0f7015d340 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f700349c0 con 0x7f0f70030b10 2023-10-30T20:24:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7015dd00 con 0x7f0f70033600 2023-10-30T20:24:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7015ce20 con 0x7f0f7015d340 2023-10-30T20:24:07.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f0f7fa2d640 10 monclient(hunting): authenticate will time out at 1570.226988s 2023-10-30T20:24:07.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7f1cf984b640 1 -- 172.21.15.69:0/3775449672 wait complete. 2023-10-30T20:24:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f1cf984b640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f1cf984b640 10 monclient: monmap: 2023-10-30T20:24:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.005 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.005 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.005 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.005 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f9bfe2dd640 1 --2- 172.21.15.69:0/2392942984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bf00070e0 0x7f9bf0035e20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00084b0 tx=0x7f9be802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cad577edde3a38e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.006 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7fb991079640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb98c13eb80 0x7fb98c13efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.006 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.937+0000 7fb99207b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb98c145360 0x7fb98c1457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.006 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7fb99207b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb98c145360 0x7fb98c1457f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45182/0 (socket says 172.21.15.69:45182) 2023-10-30T20:24:07.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7fb99207b640 1 -- 172.21.15.69:0/1914704085 learned_addr learned my addr 172.21.15.69:0/1914704085 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7fb99207b640 10 monclient(hunting): get_auth_request con 0x7fb98c145360 auth_method 0 2023-10-30T20:24:07.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7fb99207b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7fb99207b640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7fb99207b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f1cf984b640 1 librados: starting msgr at 2023-10-30T20:24:07.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.938+0000 7f1cf984b640 1 librados: starting objecter 2023-10-30T20:24:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.939+0000 7f1cf984b640 1 Processor -- start 2023-10-30T20:24:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.939+0000 7f1cf984b640 1 -- start start 2023-10-30T20:24:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.939+0000 7f1cf984b640 1 librados: setting wanted keys 2023-10-30T20:24:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.939+0000 7f1cf984b640 1 librados: calling monclient init 2023-10-30T20:24:07.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.939+0000 7f1cf984b640 10 monclient: init 2023-10-30T20:24:07.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.939+0000 7fb99187a640 1 --2- 172.21.15.69:0/1914704085 >> v2:172.21.15.183:3300/0 conn(0x7fb98c0aca50 0x7fb98c06db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.939+0000 7fb99187a640 10 monclient(hunting): get_auth_request con 0x7fb98c0aca50 auth_method 0 2023-10-30T20:24:07.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.939+0000 7fb99187a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.939+0000 7fb99187a640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.939+0000 7fb99187a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f46237fe640 1 --2- 172.21.15.69:0/307714617 >> v2:172.21.15.69:3300/0 conn(0x7f4624071af0 0x7f46240ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f4623fff640 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2023-10-30T20:24:07.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f4623fff640 1 -- 172.21.15.69:0/307714617 >> v2:172.21.15.69:3300/0 conn(0x7f4624071af0 msgr2=0x7f46240ae690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f4623fff640 1 --2- 172.21.15.69:0/307714617 >> v2:172.21.15.69:3300/0 conn(0x7f4624071af0 0x7f46240ae690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f4623fff640 1 -- 172.21.15.69:0/307714617 >> v1:172.21.15.183:6789/0 conn(0x7f4624143350 legacy=0x7f4624070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.013 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f4623fff640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.013 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f4623fff640 1 monclient: found mon.noname-b 2023-10-30T20:24:07.013 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f4623fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.013 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f4623fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f4623fff640 1 -- 172.21.15.69:0/307714617 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4624070130 con 0x7f4624071710 2023-10-30T20:24:07.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:24:07.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cec0070e0 0x7f1cec02d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 10 monclient(hunting): picked mon.noname-a con 0x7f1cec0070e0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1cec030b10 0x7f1cec0c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 10 monclient(hunting): picked mon.noname-b con 0x7f1cec030b10 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.016 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cec036530 0x7f1cec02f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.016 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.939+0000 7f0f7cfc8640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f70033600 0x7f0f700c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7fb99207b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7fb99207b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7fb99207b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 10 monclient(hunting): picked mon.noname-c con 0x7f1cec036530 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7fb99187a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7fb99187a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7fb99187a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec000be0 con 0x7f1cec0070e0 2023-10-30T20:24:07.020 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.020 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec15c8a0 con 0x7f1cec030b10 2023-10-30T20:24:07.020 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.020 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec0c12a0 con 0x7f1cec036530 2023-10-30T20:24:07.020 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.021 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f1cf984b640 10 monclient(hunting): authenticate will time out at 1570.229327s 2023-10-30T20:24:07.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f0f7cfc8640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f70033600 0x7f0f700c7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36446/0 (socket says 172.21.15.69:36446) 2023-10-30T20:24:07.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f0f7cfc8640 1 -- 172.21.15.69:0/2087719441 learned_addr learned my addr 172.21.15.69:0/2087719441 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f0f7cfc8640 10 monclient(hunting): get_auth_request con 0x7f0f70033600 auth_method 0 2023-10-30T20:24:07.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f0f7cfc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.023 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f0f7cfc8640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.023 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.940+0000 7f0f7cfc8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.023 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.941+0000 7fb991079640 10 monclient(hunting): get_auth_request con 0x7fb98c13eb80 auth_method 0 2023-10-30T20:24:07.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.941+0000 7fb991079640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.941+0000 7fb991079640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.941+0000 7fb991079640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.942+0000 7fb99187a640 10 monclient(hunting): handle_auth_done global_id 4276 payload 274 2023-10-30T20:24:07.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.942+0000 7f0f7dfca640 1 --2- 172.21.15.69:0/2087719441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f7015d340 0x7f0f7015d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.942+0000 7fb99187a640 1 -- 172.21.15.69:0/1914704085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb98c13eb80 msgr2=0x7fb98c13efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.026 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.942+0000 7fb99187a640 1 --2- 172.21.15.69:0/1914704085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb98c13eb80 0x7fb98c13efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.026 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.942+0000 7fb99187a640 1 -- 172.21.15.69:0/1914704085 >> v2:172.21.15.69:3300/0 conn(0x7fb98c145360 msgr2=0x7fb98c1457f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.026 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.942+0000 7fb99187a640 1 --2- 172.21.15.69:0/1914704085 >> v2:172.21.15.69:3300/0 conn(0x7fb98c145360 0x7fb98c1457f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.026 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.942+0000 7fb99187a640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.026 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.942+0000 7fb99187a640 1 monclient: found mon.noname-c 2023-10-30T20:24:07.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.942+0000 7fb99187a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.942+0000 7fb99187a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.942+0000 7fb99187a640 1 -- 172.21.15.69:0/1914704085 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98c13f4e0 con 0x7fb98c0aca50 2023-10-30T20:24:07.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.942+0000 7fae99621640 10 monclient: get_monmap_and_config 2023-10-30T20:24:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.942+0000 7fae99621640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.942+0000 7fae99621640 10 monclient: monmap: 2023-10-30T20:24:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.029 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:06.943645+0000 2023-10-30T20:24:07.029 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:06.943645+0000 2023-10-30T20:24:07.029 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.029 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.029 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:07.030 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:07.030 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:07.030 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:07.030 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:07.031 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.031 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.943+0000 7f0f7cfc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.031 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.943+0000 7f0f7cfc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.032 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.943+0000 7f0f7cfc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.032 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.943+0000 7fae99621640 1 Processor -- start 2023-10-30T20:24:07.032 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.943+0000 7fae99621640 1 -- start start 2023-10-30T20:24:07.033 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.943+0000 7fae99621640 10 monclient: init 2023-10-30T20:24:07.033 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.943+0000 7f0f7dfca640 10 monclient(hunting): get_auth_request con 0x7f0f7015d340 auth_method 0 2023-10-30T20:24:07.033 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.943+0000 7f0f7dfca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.033 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.943+0000 7f0f7dfca640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.034 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.943+0000 7f0f7dfca640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.035 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.944+0000 7fae99621640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.035 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.944+0000 7fae99621640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-10-30T20:24:07.035 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.944+0000 7fae99621640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae940a8e40 0x7fae9413efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.035 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.944+0000 7fae99621640 10 monclient(hunting): picked mon.noname-b con 0x7fae940a8e40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.944+0000 7fae99621640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fae940b13c0 0x7fae94141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.944+0000 7fae99621640 10 monclient(hunting): picked mon.noname-a con 0x7fae940b13c0 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:07.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.944+0000 7fae99621640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae9413f500 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:07.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.944+0000 7fae99621640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae940b1b00 con 0x7fae9413f500 2023-10-30T20:24:07.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.944+0000 7fae99621640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.944+0000 7fae99621640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae940679a0 con 0x7fae940b13c0 2023-10-30T20:24:07.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.944+0000 7fae99621640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.944+0000 7fae99621640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae940b1060 con 0x7fae940a8e40 2023-10-30T20:24:07.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.944+0000 7fae99621640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.944+0000 7fae99621640 10 monclient(hunting): authenticate will time out at 1570.233065s 2023-10-30T20:24:07.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.945+0000 7f0f7dfca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.945+0000 7f0f7dfca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.945+0000 7f0f7dfca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.945+0000 7feebe11d640 10 monclient: get_monmap_and_config 2023-10-30T20:24:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.945+0000 7feebe11d640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.945+0000 7feebe11d640 10 monclient: monmap: 2023-10-30T20:24:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:06.946712+0000 2023-10-30T20:24:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:06.946712+0000 2023-10-30T20:24:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.041 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:07.041 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:07.041 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:07.041 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:07.042 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:07.042 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.946+0000 7f0f7cfc8640 10 monclient(hunting): handle_auth_done global_id 4279 payload 824 2023-10-30T20:24:07.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.946+0000 7f0f7cfc8640 1 -- 172.21.15.69:0/2087719441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f7015d340 msgr2=0x7f0f7015d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.946+0000 7f0f7cfc8640 1 --2- 172.21.15.69:0/2087719441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f7015d340 0x7f0f7015d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.946+0000 7f0f7cfc8640 1 -- 172.21.15.69:0/2087719441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f70030b10 msgr2=0x7f0f700c6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.946+0000 7f0f7cfc8640 1 --2- 172.21.15.69:0/2087719441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f70030b10 0x7f0f700c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.946+0000 7f0f7cfc8640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.946+0000 7f0f7cfc8640 1 monclient: found mon.noname-b 2023-10-30T20:24:07.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.946+0000 7f0f7cfc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.946+0000 7f0f7cfc8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.946+0000 7f0f7cfc8640 1 -- 172.21.15.69:0/2087719441 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f700c6cb0 con 0x7f0f70033600 2023-10-30T20:24:07.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.946+0000 7feebe11d640 1 Processor -- start 2023-10-30T20:24:07.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.947+0000 7feebe11d640 1 -- start start 2023-10-30T20:24:07.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.947+0000 7feebe11d640 10 monclient: init 2023-10-30T20:24:07.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.947+0000 7feebe11d640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.947+0000 7feebe11d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-10-30T20:24:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.947+0000 7feebe11d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7feeb80a8e40 0x7feeb813efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.947+0000 7feebe11d640 10 monclient(hunting): picked mon.noname-c con 0x7feeb80a8e40 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.947+0000 7feebe11d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7feeb80b13c0 0x7feeb8141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.947+0000 7feebe11d640 10 monclient(hunting): picked mon.noname-a con 0x7feeb80b13c0 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:07.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.947+0000 7feebe11d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feeb813f500 0x7feeb81423f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.947+0000 7feebe11d640 10 monclient(hunting): picked mon.noname-b con 0x7feeb813f500 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.947+0000 7feebe11d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.947+0000 7feebe11d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeb80679a0 con 0x7feeb80b13c0 2023-10-30T20:24:07.050 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.947+0000 7feebe11d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.050 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.947+0000 7feebe11d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeb80b1060 con 0x7feeb80a8e40 2023-10-30T20:24:07.050 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.948+0000 7feebe11d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.050 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.948+0000 7feebe11d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb80a9220 con 0x7feeb813f500 2023-10-30T20:24:07.050 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.948+0000 7feebe11d640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.051 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.948+0000 7feebe11d640 10 monclient(hunting): authenticate will time out at 1570.236867s 2023-10-30T20:24:07.051 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.949+0000 7f1cf2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cec0070e0 0x7f1cec02d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.949+0000 7f0f6dffb640 10 client.?.objecter ms_handle_connect 0x7f0f70033600 2023-10-30T20:24:07.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.949+0000 7f0f6dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:07.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.949+0000 7f0f7cfc8640 1 --2- 172.21.15.69:0/2087719441 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f70033600 0x7f0f700c7480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0f6800e9b0 tx=0x7f0f6800edb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.949+0000 7f1cf2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cec0070e0 0x7f1cec02d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45204/0 (socket says 172.21.15.69:45204) 2023-10-30T20:24:07.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.949+0000 7f1cf2ffd640 1 -- 172.21.15.69:0/3287870425 learned_addr learned my addr 172.21.15.69:0/3287870425 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.949+0000 7f1cf2ffd640 10 monclient(hunting): get_auth_request con 0x7f1cec0070e0 auth_method 0 2023-10-30T20:24:07.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.949+0000 7f1cf2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.949+0000 7f1cf2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.949+0000 7f1cf2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.949+0000 7fae927fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fae940b13c0 0x7fae94141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7f1cf2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7f1cf2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7f1cf2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7f9bfd2db640 1 -- 172.21.15.69:0/2392942984 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be80315a0 con 0x7f9bf00070e0 2023-10-30T20:24:07.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7f9bfd2db640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7f9bfd2db640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:07.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7f9bfd2db640 10 monclient: dump: 2023-10-30T20:24:07.057 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.057 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.057 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.057 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.058 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.058 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.058 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.058 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.058 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.059 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7f9bfd2db640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7f9bfd2db640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7f9bfd2db640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7fae927fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fae940b13c0 0x7fae94141eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45220/0 (socket says 172.21.15.69:45220) 2023-10-30T20:24:07.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7fae927fc640 1 -- 172.21.15.69:0/194359510 learned_addr learned my addr 172.21.15.69:0/194359510 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7fae927fc640 10 monclient(hunting): get_auth_request con 0x7fae940b13c0 auth_method 0 2023-10-30T20:24:07.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7fae927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7fae927fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7f9c00541640 5 monclient: authenticate success, global_id 4273 2023-10-30T20:24:07.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7f9c00541640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:07.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.950+0000 7fae927fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.951+0000 7f1cf37fe640 1 --2- 172.21.15.69:0/3287870425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cec036530 0x7f1cec02f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.952+0000 7f592431d640 10 monclient: get_monmap_and_config 2023-10-30T20:24:07.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.952+0000 7f592431d640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.952+0000 7f592431d640 10 monclient: monmap: 2023-10-30T20:24:07.063 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.063 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.063 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:06.953227+0000 2023-10-30T20:24:07.064 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:06.953227+0000 2023-10-30T20:24:07.064 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.064 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.064 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:07.064 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:07.064 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:07.065 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.952+0000 7f592431d640 1 Processor -- start 2023-10-30T20:24:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.952+0000 7f592431d640 1 -- start start 2023-10-30T20:24:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.952+0000 7f592431d640 10 monclient: init 2023-10-30T20:24:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f592431d640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f1cf37fe640 10 monclient(hunting): get_auth_request con 0x7f1cec036530 auth_method 0 2023-10-30T20:24:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f592431d640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2023-10-30T20:24:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f1cf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f1cf37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f1cf37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f592431d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f591c0a8e40 addr v1:172.21.15.69:6789/0 2023-10-30T20:24:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f592431d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f591c0b13c0 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f592431d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f591c1425c0 0x7f591c14d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f1cf2ffd640 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2023-10-30T20:24:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f592431d640 10 monclient(hunting): picked mon.noname-a con 0x7f591c1425c0 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f592431d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591c0b1b00 con 0x7f591c0a8e40 2023-10-30T20:24:07.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f592431d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591c1429d0 con 0x7f591c0b13c0 2023-10-30T20:24:07.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f592431d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f592431d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f591c0679a0 con 0x7f591c1425c0 2023-10-30T20:24:07.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f592431d640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f592431d640 10 monclient(hunting): authenticate will time out at 1570.242747s 2023-10-30T20:24:07.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f1cf2ffd640 1 -- 172.21.15.69:0/3287870425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cec036530 msgr2=0x7f1cec02f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f1cf2ffd640 1 --2- 172.21.15.69:0/3287870425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cec036530 0x7f1cec02f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f1cf2ffd640 1 -- 172.21.15.69:0/3287870425 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1cec030b10 msgr2=0x7f1cec0c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f1cf2ffd640 1 --2- 172.21.15.69:0/3287870425 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1cec030b10 0x7f1cec0c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f1cf2ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f1cf2ffd640 1 monclient: found mon.noname-a 2023-10-30T20:24:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f1cf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f1cf2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.953+0000 7f1cf2ffd640 1 -- 172.21.15.69:0/3287870425 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec15cb40 con 0x7f1cec0070e0 2023-10-30T20:24:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7f9bfd2db640 1 -- 172.21.15.69:0/2392942984 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be8031710 con 0x7f9bf00070e0 2023-10-30T20:24:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7f9bfd2db640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7f9bfd2db640 1 -- 172.21.15.69:0/2392942984 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be8031af0 con 0x7f9bf00070e0 2023-10-30T20:24:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7f9bfd2db640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7f9bfd2db640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7f9bfd2db640 10 monclient: dump: 2023-10-30T20:24:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.075 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.075 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.075 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.075 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.076 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.076 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.076 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.076 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7fae927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7fae927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7fae927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7f9c00541640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:07.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7f9c00541640 4 set_mon_vals no callback set 2023-10-30T20:24:07.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7feeb77fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7feeb80a8e40 0x7feeb813efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7f4623fff640 1 --2- 172.21.15.69:0/307714617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4624071710 0x7f4624145df0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4618009870 tx=0x7f461802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=936d5aef93273a7f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7f9c00541640 10 monclient: shutdown 2023-10-30T20:24:07.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7f9c00541640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7f9c00541640 1 -- 172.21.15.69:0/2392942984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bf00070e0 msgr2=0x7f9bf0035e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.954+0000 7f9c00541640 1 --2- 172.21.15.69:0/2392942984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bf00070e0 0x7f9bf0035e20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00084b0 tx=0x7f9be802b840 comp rx=0 tx=0).stop 2023-10-30T20:24:07.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.955+0000 7f9c00541640 1 -- 172.21.15.69:0/2392942984 shutdown_connections 2023-10-30T20:24:07.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.955+0000 7f9c00541640 1 --2- 172.21.15.69:0/2392942984 >> v2:172.21.15.69:3300/0 conn(0x7f9bf0033d10 0x7f9bf0036360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.955+0000 7f9c00541640 1 --2- 172.21.15.69:0/2392942984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bf00070e0 0x7f9bf0035e20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.955+0000 7f9c00541640 1 -- 172.21.15.69:0/2392942984 >> 172.21.15.69:0/2392942984 conn(0x7f9bf0006030 msgr2=0x7f9bf0006440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.955+0000 7f9c00541640 1 -- 172.21.15.69:0/2392942984 shutdown_connections 2023-10-30T20:24:07.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.955+0000 7f9c00541640 1 -- 172.21.15.69:0/2392942984 wait complete. 2023-10-30T20:24:07.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.955+0000 7f9c00541640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.955+0000 7f9c00541640 10 monclient: monmap: 2023-10-30T20:24:07.082 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.083 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.083 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.083 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.083 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.083 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.083 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.084 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.084 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.084 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 1 librados: starting msgr at 2023-10-30T20:24:07.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 1 librados: starting objecter 2023-10-30T20:24:07.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 1 Processor -- start 2023-10-30T20:24:07.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 1 -- start start 2023-10-30T20:24:07.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 1 librados: setting wanted keys 2023-10-30T20:24:07.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 1 librados: calling monclient init 2023-10-30T20:24:07.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 10 monclient: init 2023-10-30T20:24:07.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:24:07.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bf00070e0 0x7f9bf002fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 10 monclient(hunting): picked mon.noname-b con 0x7f9bf00070e0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bf0030b10 0x7f9bf002d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf0030b10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bf0033d10 0x7f9bf00c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf0033d10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf0005e80 con 0x7f9bf0033d10 2023-10-30T20:24:07.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf00c7ec0 con 0x7f9bf00070e0 2023-10-30T20:24:07.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf00c6630 con 0x7f9bf0030b10 2023-10-30T20:24:07.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.956+0000 7f9c00541640 10 monclient(hunting): authenticate will time out at 1570.245571s 2023-10-30T20:24:07.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.957+0000 7feeb77fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7feeb80a8e40 0x7feeb813efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36470/0 (socket says 172.21.15.69:36470) 2023-10-30T20:24:07.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.957+0000 7feeb77fe640 1 -- 172.21.15.69:0/689861333 learned_addr learned my addr 172.21.15.69:0/689861333 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.957+0000 7feeb77fe640 10 monclient(hunting): get_auth_request con 0x7feeb80a8e40 auth_method 0 2023-10-30T20:24:07.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.957+0000 7feeb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.957+0000 7feeb77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.957+0000 7feeb77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.957+0000 7feeb7fff640 1 --2- 172.21.15.69:0/689861333 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feeb813f500 0x7feeb81423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.958+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.958+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.958+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.959+0000 7fae927fc640 10 monclient(hunting): handle_auth_done global_id 4287 payload 274 2023-10-30T20:24:07.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.959+0000 7fae927fc640 1 -- 172.21.15.69:0/194359510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae940a8e40 msgr2=0x7fae9413efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.959+0000 7fae927fc640 1 --2- 172.21.15.69:0/194359510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae940a8e40 0x7fae9413efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.959+0000 7fae927fc640 1 -- 172.21.15.69:0/194359510 >> v1:172.21.15.183:6789/0 conn(0x7fae9413f500 legacy=0x7fae941423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.959+0000 7fae927fc640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.959+0000 7fae927fc640 1 monclient: found mon.noname-a 2023-10-30T20:24:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.959+0000 7fae927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.959+0000 7fae927fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.959+0000 7fae927fc640 1 -- 172.21.15.69:0/194359510 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae940a8920 con 0x7fae940b13c0 2023-10-30T20:24:07.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f1cf2ffd640 1 --2- 172.21.15.69:0/3287870425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cec0070e0 0x7f1cec02d7f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400c480 tx=0x7f1ce400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f1cdbfff640 10 client.?.objecter ms_handle_connect 0x7f1cec0070e0 2023-10-30T20:24:07.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f1cdbfff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:07.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f6dffb640 1 -- 172.21.15.69:0/2087719441 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6800c940 con 0x7f0f70033600 2023-10-30T20:24:07.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f6dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:07.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f6dffb640 10 monclient: dump: 2023-10-30T20:24:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.097 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.097 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.097 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.097 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.097 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f6dffb640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f6dffb640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f6dffb640 1 -- 172.21.15.69:0/2087719441 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6800d040 con 0x7f0f70033600 2023-10-30T20:24:07.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f6dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f6dffb640 1 -- 172.21.15.69:0/2087719441 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6800ccf0 con 0x7f0f70033600 2023-10-30T20:24:07.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f6dffb640 10 monclient: dump: 2023-10-30T20:24:07.099 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.099 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.099 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.100 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.100 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.100 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.100 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.100 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.100 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.101 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f7fa2d640 5 monclient: authenticate success, global_id 4279 2023-10-30T20:24:07.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f7fa2d640 10 monclient: _renew_subs 2023-10-30T20:24:07.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f7fa2d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f6f7fe640 4 set_mon_vals no callback set 2023-10-30T20:24:07.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.960+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2087719441 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f700c79c0 con 0x7f0f70033600 2023-10-30T20:24:07.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7f0f7fa2d640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7f0f7fa2d640 10 monclient: _renew_subs 2023-10-30T20:24:07.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7f0f7fa2d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2087719441 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7002ea50 con 0x7f0f70033600 2023-10-30T20:24:07.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7f0f7fa2d640 1 librados: init done 2023-10-30T20:24:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7feeb7fff640 10 monclient(hunting): get_auth_request con 0x7feeb813f500 auth_method 0 2023-10-30T20:24:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7feeb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7feeb7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7feeb7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7f0f7fa2d640 10 client.4279.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7f0f7fa2d640 15 client.4279.objecter _session_command_op_assign -1 1 2023-10-30T20:24:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7f0f7fa2d640 20 client.4279.objecter _get_session osd=-1 returning homeless 2023-10-30T20:24:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7f0f7fa2d640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7f0f7fa2d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.961+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2087719441 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0f70035890 con 0x7f0f70033600 2023-10-30T20:24:07.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.962+0000 7feeb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.962+0000 7feeb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.962+0000 7feeb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.962+0000 7f4622ffd640 1 -- 172.21.15.69:0/307714617 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46180317c0 con 0x7f4624071710 2023-10-30T20:24:07.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.962+0000 7f4622ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.962+0000 7fae927fc640 1 --2- 172.21.15.69:0/194359510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae940b13c0 0x7fae94141eb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fae88009870 tx=0x7fae8802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bfab4a680eb22a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.962+0000 7f4622ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:07.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.962+0000 7f4622ffd640 10 monclient: dump: 2023-10-30T20:24:07.109 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.109 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.109 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.110 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.110 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.110 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.110 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.110 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.110 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.111 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f4622ffd640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f4622ffd640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f4622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f462a7f5640 5 monclient: authenticate success, global_id 4256 2023-10-30T20:24:07.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f462a7f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:07.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f4622ffd640 1 -- 172.21.15.69:0/307714617 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461802e070 con 0x7f4624071710 2023-10-30T20:24:07.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f4622ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f462a7f5640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:07.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f462a7f5640 4 set_mon_vals no callback set 2023-10-30T20:24:07.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f4622ffd640 1 -- 172.21.15.69:0/307714617 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4618031b50 con 0x7f4624071710 2023-10-30T20:24:07.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f4622ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f4622ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:07.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f4622ffd640 10 monclient: dump: 2023-10-30T20:24:07.113 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.113 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.114 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.114 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.114 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.114 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.114 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.115 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.115 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.115 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.115 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f462a7f5640 10 monclient: shutdown 2023-10-30T20:24:07.115 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f462a7f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.115 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f462a7f5640 1 -- 172.21.15.69:0/307714617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4624071710 msgr2=0x7f4624145df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f462a7f5640 1 --2- 172.21.15.69:0/307714617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4624071710 0x7f4624145df0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4618009870 tx=0x7f461802b860 comp rx=0 tx=0).stop 2023-10-30T20:24:07.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f462a7f5640 1 -- 172.21.15.69:0/307714617 shutdown_connections 2023-10-30T20:24:07.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f462a7f5640 1 --2- 172.21.15.69:0/307714617 >> v2:172.21.15.69:3300/0 conn(0x7f4624071af0 0x7f46240ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f462a7f5640 1 --2- 172.21.15.69:0/307714617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4624071710 0x7f4624145df0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f462a7f5640 1 -- 172.21.15.69:0/307714617 >> 172.21.15.69:0/307714617 conn(0x7f4624069d50 msgr2=0x7f462406a130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7feeb77fe640 10 monclient(hunting): handle_auth_done global_id 4282 payload 274 2023-10-30T20:24:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7feeb77fe640 1 -- 172.21.15.69:0/689861333 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feeb813f500 msgr2=0x7feeb81423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7feeb77fe640 1 --2- 172.21.15.69:0/689861333 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feeb813f500 0x7feeb81423f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7feeb77fe640 1 -- 172.21.15.69:0/689861333 >> v2:172.21.15.69:3300/0 conn(0x7feeb80b13c0 msgr2=0x7feeb8141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.118 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7feeb77fe640 1 --2- 172.21.15.69:0/689861333 >> v2:172.21.15.69:3300/0 conn(0x7feeb80b13c0 0x7feeb8141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.118 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7feeb77fe640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.118 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7feeb77fe640 1 monclient: found mon.noname-c 2023-10-30T20:24:07.118 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7feeb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.118 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7feeb77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.119 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7feeb77fe640 1 -- 172.21.15.69:0/689861333 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb80a8920 con 0x7feeb80a8e40 2023-10-30T20:24:07.120 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f462a7f5640 1 -- 172.21.15.69:0/307714617 shutdown_connections 2023-10-30T20:24:07.120 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.963+0000 7f462a7f5640 1 -- 172.21.15.69:0/307714617 wait complete. 2023-10-30T20:24:07.120 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.964+0000 7f462a7f5640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.964+0000 7f462a7f5640 10 monclient: monmap: 2023-10-30T20:24:07.121 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.121 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.121 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.121 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.121 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.122 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.122 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.122 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.122 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.122 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.123 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.964+0000 7f462a7f5640 1 librados: starting msgr at 2023-10-30T20:24:07.123 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.964+0000 7f462a7f5640 1 librados: starting objecter 2023-10-30T20:24:07.123 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.964+0000 7f462a7f5640 1 Processor -- start 2023-10-30T20:24:07.123 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 1 -- start start 2023-10-30T20:24:07.123 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 1 librados: setting wanted keys 2023-10-30T20:24:07.123 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 1 librados: calling monclient init 2023-10-30T20:24:07.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 10 monclient: init 2023-10-30T20:24:07.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T20:24:07.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4624071710 0x7f46240aa840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 10 monclient(hunting): picked mon.noname-c con 0x7f4624071710 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4624143350 0x7f46240abf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 10 monclient(hunting): picked mon.noname-b con 0x7f4624143350 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46240ac4a0 0x7f46240ac930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 10 monclient(hunting): picked mon.noname-a con 0x7f46240ac4a0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462406ab30 con 0x7f46240ac4a0 2023-10-30T20:24:07.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46240aad80 con 0x7f4624143350 2023-10-30T20:24:07.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4624147ce0 con 0x7f4624071710 2023-10-30T20:24:07.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.965+0000 7f462a7f5640 10 monclient(hunting): authenticate will time out at 1570.254495s 2023-10-30T20:24:07.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.966+0000 7f59218b8640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f591c0b13c0 0x7f591c141eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.69:33164/0 (socket says 172.21.15.69:33164) 2023-10-30T20:24:07.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.966+0000 7f59218b8640 1 -- 172.21.15.69:0/2027567513 learned_addr learned my addr 172.21.15.69:0/2027567513 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:24:07.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.969+0000 7f9bfdadc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bf0030b10 0x7f9bf002d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.969+0000 7f9bfdadc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bf0030b10 0x7f9bf002d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60674/0 (socket says 172.21.15.69:60674) 2023-10-30T20:24:07.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.969+0000 7f9bfdadc640 1 -- 172.21.15.69:0/1950889256 learned_addr learned my addr 172.21.15.69:0/1950889256 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.969+0000 7f9bfdadc640 10 monclient(hunting): get_auth_request con 0x7f9bf0030b10 auth_method 0 2023-10-30T20:24:07.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.969+0000 7f9bfdadc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.969+0000 7f9bfdadc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.969+0000 7f9bfdadc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cdbfff640 1 -- 172.21.15.69:0/3287870425 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce40052d0 con 0x7f1cec0070e0 2023-10-30T20:24:07.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cdbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:07.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cdbfff640 10 monclient: dump: 2023-10-30T20:24:07.131 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.131 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.131 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.131 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.131 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.132 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.132 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.132 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.132 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.132 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.132 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cdbfff640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cdbfff640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cf984b640 5 monclient: authenticate success, global_id 4284 2023-10-30T20:24:07.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cf984b640 10 monclient: _renew_subs 2023-10-30T20:24:07.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cf984b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cf984b640 1 -- 172.21.15.69:0/3287870425 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cec0c7300 con 0x7f1cec0070e0 2023-10-30T20:24:07.134 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cdbfff640 1 -- 172.21.15.69:0/3287870425 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce400e040 con 0x7f1cec0070e0 2023-10-30T20:24:07.134 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cdbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.134 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cf984b640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.134 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cdbfff640 1 -- 172.21.15.69:0/3287870425 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4012630 con 0x7f1cec0070e0 2023-10-30T20:24:07.134 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cdbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:07.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cdbfff640 10 monclient: dump: 2023-10-30T20:24:07.135 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.135 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.135 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.136 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.136 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.136 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.136 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.136 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.136 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.137 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cf984b640 10 monclient: _renew_subs 2023-10-30T20:24:07.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cf984b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cf984b640 1 -- 172.21.15.69:0/3287870425 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cec15d1b0 con 0x7f1cec0070e0 2023-10-30T20:24:07.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cf984b640 1 librados: init done 2023-10-30T20:24:07.138 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f9bfdadc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.138 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f9bfdadc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.138 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f9bfdadc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.139 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f59228ba640 1 --2- 172.21.15.69:0/2027567513 >> v2:172.21.15.69:3300/0 conn(0x7f591c1425c0 0x7f591c14d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.139 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.970+0000 7f1cf17fa640 4 set_mon_vals no callback set 2023-10-30T20:24:07.140 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.971+0000 7f59228ba640 10 monclient(hunting): get_auth_request con 0x7f591c1425c0 auth_method 0 2023-10-30T20:24:07.140 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.971+0000 7f59228ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.140 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.971+0000 7f59228ba640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.140 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.971+0000 7f59228ba640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.971+0000 7f1cf984b640 10 client.4284.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:07.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.971+0000 7f1cf984b640 15 client.4284.objecter _session_command_op_assign -1 1 2023-10-30T20:24:07.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.971+0000 7f1cf984b640 20 client.4284.objecter _get_session osd=-1 returning homeless 2023-10-30T20:24:07.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.971+0000 7f1cf984b640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.971+0000 7f1cf984b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.971+0000 7f1cf984b640 1 -- 172.21.15.69:0/3287870425 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1cec0c56d0 con 0x7f1cec0070e0 2023-10-30T20:24:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.972+0000 7f59228ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.143 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.972+0000 7f59228ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.143 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.972+0000 7f59228ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.973+0000 7f9bfdadc640 10 monclient(hunting): handle_auth_done global_id 4271 payload 824 2023-10-30T20:24:07.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.973+0000 7f9bfdadc640 1 -- 172.21.15.69:0/1950889256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bf00070e0 msgr2=0x7f9bf002fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.973+0000 7f9bfdadc640 1 --2- 172.21.15.69:0/1950889256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bf00070e0 0x7f9bf002fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.973+0000 7f9bfdadc640 1 -- 172.21.15.69:0/1950889256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bf0033d10 msgr2=0x7f9bf00c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.973+0000 7f9bfdadc640 1 --2- 172.21.15.69:0/1950889256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bf0033d10 0x7f9bf00c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.973+0000 7f9bfdadc640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.973+0000 7f9bfdadc640 1 monclient: found mon.noname-c 2023-10-30T20:24:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.973+0000 7f9bfdadc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.973+0000 7f9bfdadc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.973+0000 7f9bfdadc640 1 -- 172.21.15.69:0/1950889256 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf00c7980 con 0x7f9bf0030b10 2023-10-30T20:24:07.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.974+0000 7f59228ba640 10 monclient(hunting): handle_auth_done global_id 4290 payload 274 2023-10-30T20:24:07.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.974+0000 7f59228ba640 1 -- 172.21.15.69:0/2027567513 >> v1:172.21.15.183:6789/0 conn(0x7f591c0b13c0 legacy=0x7f591c141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.974+0000 7f59228ba640 1 -- 172.21.15.69:0/2027567513 >> v1:172.21.15.69:6789/0 conn(0x7f591c0a8e40 legacy=0x7f591c13efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.974+0000 7f59228ba640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.974+0000 7f59228ba640 1 monclient: found mon.noname-a 2023-10-30T20:24:07.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.974+0000 7f59228ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.974+0000 7f59228ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.974+0000 7f59228ba640 1 -- 172.21.15.69:0/2027567513 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c0a8920 con 0x7f591c1425c0 2023-10-30T20:24:07.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.976+0000 7fb99187a640 1 --2- 172.21.15.69:0/1914704085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98c0aca50 0x7fb98c06db60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb98c0ad090 tx=0x7fb98802b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6a792142c33d20a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.976+0000 7f46237fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4624143350 0x7f46240abf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.976+0000 7f9bfdadc640 1 --2- 172.21.15.69:0/1950889256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bf0030b10 0x7f9bf002d580 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400c560 tx=0x7f9bf400ca20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.976+0000 7f9beeffd640 10 client.?.objecter ms_handle_connect 0x7f9bf0030b10 2023-10-30T20:24:07.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.976+0000 7f9beeffd640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:07.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.979+0000 7f4623fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4624071710 0x7f46240aa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.979+0000 7f46237fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4624143350 0x7f46240abf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36488/0 (socket says 172.21.15.69:36488) 2023-10-30T20:24:07.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.979+0000 7f46237fe640 1 -- 172.21.15.69:0/3841363599 learned_addr learned my addr 172.21.15.69:0/3841363599 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.979+0000 7f46237fe640 10 monclient(hunting): get_auth_request con 0x7f4624143350 auth_method 0 2023-10-30T20:24:07.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.979+0000 7f46237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.979+0000 7f46237fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.979+0000 7f46237fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.979+0000 7f4623fff640 10 monclient(hunting): get_auth_request con 0x7f4624071710 auth_method 0 2023-10-30T20:24:07.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.979+0000 7f4623fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.979+0000 7f4623fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.979+0000 7f4623fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.980+0000 7f4623fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.980+0000 7f4623fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.980+0000 7f4623fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.981+0000 7f46237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.981+0000 7f46237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.981+0000 7f46237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.982+0000 7f4623fff640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2023-10-30T20:24:07.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.982+0000 7f4623fff640 1 -- 172.21.15.69:0/3841363599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4624143350 msgr2=0x7f46240abf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.982+0000 7f4623fff640 1 --2- 172.21.15.69:0/3841363599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4624143350 0x7f46240abf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.983+0000 7f4623fff640 1 -- 172.21.15.69:0/3841363599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46240ac4a0 msgr2=0x7f46240ac930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.983+0000 7f4623fff640 1 --2- 172.21.15.69:0/3841363599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46240ac4a0 0x7f46240ac930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.983+0000 7f4623fff640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.983+0000 7f4623fff640 1 monclient: found mon.noname-c 2023-10-30T20:24:07.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.983+0000 7f4623fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.983+0000 7f4623fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.157 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.983+0000 7f4623fff640 1 -- 172.21.15.69:0/3841363599 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4624147f30 con 0x7f4624071710 2023-10-30T20:24:07.157 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.984+0000 7f59228ba640 1 --2- 172.21.15.69:0/2027567513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f591c1425c0 0x7f591c14d330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5914009d90 tx=0x7f591402bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c181e604e84421c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.157 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.985+0000 7f4623fff640 1 --2- 172.21.15.69:0/3841363599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4624071710 0x7f46240aa840 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4618033000 tx=0x7f46180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.985+0000 7f4620ff9640 10 client.?.objecter ms_handle_connect 0x7f4624071710 2023-10-30T20:24:07.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.985+0000 7f4620ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:07.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9beeffd640 1 -- 172.21.15.69:0/1950889256 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4005ce0 con 0x7f9bf0030b10 2023-10-30T20:24:07.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9beeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.160 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9beeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:07.160 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9beeffd640 10 monclient: dump: 2023-10-30T20:24:07.160 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.160 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.160 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.161 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.161 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.161 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.161 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.161 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.161 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.162 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9beeffd640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9beeffd640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9beeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9c00541640 5 monclient: authenticate success, global_id 4271 2023-10-30T20:24:07.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9c00541640 10 monclient: _renew_subs 2023-10-30T20:24:07.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9c00541640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9c00541640 1 -- 172.21.15.69:0/1950889256 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf00c8350 con 0x7f9bf0030b10 2023-10-30T20:24:07.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9c00541640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9c00541640 10 monclient: _renew_subs 2023-10-30T20:24:07.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9c00541640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9c00541640 1 -- 172.21.15.69:0/1950889256 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf00c8870 con 0x7f9bf0030b10 2023-10-30T20:24:07.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.988+0000 7f9c00541640 1 librados: init done 2023-10-30T20:24:07.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.989+0000 7f9beeffd640 1 -- 172.21.15.69:0/1950889256 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf400e040 con 0x7f9bf0030b10 2023-10-30T20:24:07.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.989+0000 7f9beeffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.989+0000 7f9beeffd640 1 -- 172.21.15.69:0/1950889256 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4007270 con 0x7f9bf0030b10 2023-10-30T20:24:07.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.989+0000 7f9beeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.989+0000 7f9beeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:07.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.989+0000 7f9beeffd640 10 monclient: dump: 2023-10-30T20:24:07.165 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.165 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.165 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.166 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.166 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.166 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.166 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.166 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.167 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.167 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.989+0000 7f9bfcada640 4 set_mon_vals no callback set 2023-10-30T20:24:07.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.989+0000 7f9c00541640 10 client.4271.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:07.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.989+0000 7f9c00541640 15 client.4271.objecter _session_command_op_assign -1 1 2023-10-30T20:24:07.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.989+0000 7f9c00541640 20 client.4271.objecter _get_session osd=-1 returning homeless 2023-10-30T20:24:07.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.989+0000 7f9c00541640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.989+0000 7f9c00541640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.989+0000 7f9c00541640 1 -- 172.21.15.69:0/1950889256 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9be00028a0 con 0x7f9bf0030b10 2023-10-30T20:24:07.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.995+0000 7f0f6dffb640 1 -- 172.21.15.69:0/2087719441 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f0f68013470 con 0x7f0f70033600 2023-10-30T20:24:07.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.995+0000 7f0f6dffb640 1 --2- 172.21.15.69:0/2087719441 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0f5c06c7c0 0x7f0f5c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.995+0000 7f0f7d7c9640 1 --2- 172.21.15.69:0/2087719441 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0f5c06c7c0 0x7f0f5c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.995+0000 7f0f7d7c9640 10 monclient: get_auth_request con 0x7f0f5c06c7c0 auth_method 0 2023-10-30T20:24:07.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.996+0000 7f0f6dffb640 10 client.4279.objecter ms_handle_connect 0x7f0f5c06c7c0 2023-10-30T20:24:07.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.996+0000 7f0f7d7c9640 1 --2- 172.21.15.69:0/2087719441 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0f5c06c7c0 0x7f0f5c06ec80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0f74036000 tx=0x7f0f7403b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 1 -- 172.21.15.69:0/2087719441 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f0f680588d0 con 0x7f0f70033600 2023-10-30T20:24:07.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 10 client.4279.objecter ms_dispatch 0x7f0f78000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:07.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 3 client.4279.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:07.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 3 client.4279.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:07.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 10 client.4279.objecter checking command 1 2023-10-30T20:24:07.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 1 --2- 172.21.15.69:0/2087719441 >> v2:172.21.15.183:6804/1046221745 conn(0x7f0f5c0725d0 0x7f0f5c0749f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 20 client.4279.objecter _get_session s=0x7f0f6800ccf0 osd=6 3 2023-10-30T20:24:07.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 20 client.4279.objecter put_session s=0x7f0f6800ccf0 osd=6 3 2023-10-30T20:24:07.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 15 client.4279.objecter _session_command_op_remove -1 1 2023-10-30T20:24:07.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 20 client.4279.objecter _get_session s=0x7f0f6800ccf0 osd=6 3 2023-10-30T20:24:07.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 20 client.4279.objecter get_session s=0x7f0f6800ccf0 osd=6 3 2023-10-30T20:24:07.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 15 client.4279.objecter _session_command_op_assign 6 1 2023-10-30T20:24:07.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 20 client.4279.objecter put_session s=0x7f0f6800ccf0 osd=6 4 2023-10-30T20:24:07.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 10 client.4279.objecter _send_command 1 2023-10-30T20:24:07.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 1 -- 172.21.15.69:0/2087719441 --> v2:172.21.15.183:6804/1046221745 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f6800c940 con 0x7f0f5c0725d0 2023-10-30T20:24:07.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 20 client.4279.objecter dump_active .. 0 homeless 2023-10-30T20:24:07.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 1 -- 172.21.15.69:0/2087719441 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f68009670 con 0x7f0f70033600 2023-10-30T20:24:07.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f6dffb640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T20:24:07.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7feeb77fe640 1 --2- 172.21.15.69:0/689861333 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feeb80a8e40 0x7feeb813efc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feeb80b8030 tx=0x7feea802b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=363ce3bed6254d25 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.997+0000 7f0f7dfca640 1 --2- 172.21.15.69:0/2087719441 >> v2:172.21.15.183:6804/1046221745 conn(0x7f0f5c0725d0 0x7f0f5c0749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f0f7dfca640 10 monclient: get_auth_request con 0x7f0f5c0725d0 auth_method 0 2023-10-30T20:24:07.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f4620ff9640 1 -- 172.21.15.69:0/3841363599 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4618032860 con 0x7f4624071710 2023-10-30T20:24:07.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f4620ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f4620ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:07.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f4620ff9640 10 monclient: dump: 2023-10-30T20:24:07.177 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.177 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.177 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.178 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.178 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.178 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.178 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.178 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.179 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.179 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f4620ff9640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f4620ff9640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f4620ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f462a7f5640 5 monclient: authenticate success, global_id 4274 2023-10-30T20:24:07.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f462a7f5640 10 monclient: _renew_subs 2023-10-30T20:24:07.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f462a7f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f462a7f5640 1 -- 172.21.15.69:0/3841363599 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4624148210 con 0x7f4624071710 2023-10-30T20:24:07.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f462a7f5640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f462a7f5640 10 monclient: _renew_subs 2023-10-30T20:24:07.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f462a7f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f462a7f5640 1 -- 172.21.15.69:0/3841363599 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46240ab320 con 0x7f4624071710 2023-10-30T20:24:07.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f462a7f5640 1 librados: init done 2023-10-30T20:24:07.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f4620ff9640 1 -- 172.21.15.69:0/3841363599 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461803f030 con 0x7f4624071710 2023-10-30T20:24:07.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f4620ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f4620ff9640 1 -- 172.21.15.69:0/3841363599 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4618032c30 con 0x7f4624071710 2023-10-30T20:24:07.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f4620ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f4620ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:07.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f4620ff9640 10 monclient: dump: 2023-10-30T20:24:07.182 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.182 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.183 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.183 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.183 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.183 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.183 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.183 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.184 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.184 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.998+0000 7f46227fc640 4 set_mon_vals no callback set 2023-10-30T20:24:07.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.999+0000 7f1cdbfff640 1 -- 172.21.15.69:0/3287870425 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f1ce40127a0 con 0x7f1cec0070e0 2023-10-30T20:24:07.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.999+0000 7f462a7f5640 10 client.4274.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:07.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.999+0000 7f462a7f5640 15 client.4274.objecter _session_command_op_assign -1 1 2023-10-30T20:24:07.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.999+0000 7f462a7f5640 20 client.4274.objecter _get_session osd=-1 returning homeless 2023-10-30T20:24:07.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.999+0000 7f462a7f5640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.999+0000 7f462a7f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.999+0000 7f462a7f5640 1 -- 172.21.15.69:0/3841363599 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f46040028a0 con 0x7f4624071710 2023-10-30T20:24:07.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:06.999+0000 7f1cdbfff640 1 --2- 172.21.15.69:0/3287870425 >> v2:172.21.15.69:6800/1876307353 conn(0x7f1ce006c810 0x7f1ce006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7f1cf27fc640 1 --2- 172.21.15.69:0/3287870425 >> v2:172.21.15.69:6800/1876307353 conn(0x7f1ce006c810 0x7f1ce006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7f1cf27fc640 10 monclient: get_auth_request con 0x7f1ce006c810 auth_method 0 2023-10-30T20:24:07.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb990878640 1 -- 172.21.15.69:0/1914704085 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb988031520 con 0x7fb98c0aca50 2023-10-30T20:24:07.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb990878640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb990878640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:07.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb990878640 10 monclient: dump: 2023-10-30T20:24:07.188 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.189 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.189 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.189 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.189 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.189 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.189 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.190 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.190 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.190 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb990878640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb990878640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb990878640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb990878640 1 -- 172.21.15.69:0/1914704085 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98802f050 con 0x7fb98c0aca50 2023-10-30T20:24:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb990878640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb990878640 1 -- 172.21.15.69:0/1914704085 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9880318b0 con 0x7fb98c0aca50 2023-10-30T20:24:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb990878640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb990878640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb990878640 10 monclient: dump: 2023-10-30T20:24:07.192 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.192 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.192 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.192 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.192 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.193 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb993ade640 5 monclient: authenticate success, global_id 4276 2023-10-30T20:24:07.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb993ade640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:07.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7fb993ade640 4 set_mon_vals no callback set 2023-10-30T20:24:07.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7f1cdbfff640 10 client.4284.objecter ms_handle_connect 0x7f1ce006c810 2023-10-30T20:24:07.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.000+0000 7f1cf27fc640 1 --2- 172.21.15.69:0/3287870425 >> v2:172.21.15.69:6800/1876307353 conn(0x7f1ce006c810 0x7f1ce006ecd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1cec031280 tx=0x7f1cf40aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7fb993ade640 10 monclient: shutdown 2023-10-30T20:24:07.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7fb993ade640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7fb993ade640 1 -- 172.21.15.69:0/1914704085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98c0aca50 msgr2=0x7fb98c06db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7fb993ade640 1 --2- 172.21.15.69:0/1914704085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98c0aca50 0x7fb98c06db60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb98c0ad090 tx=0x7fb98802b7c0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7fb993ade640 1 -- 172.21.15.69:0/1914704085 shutdown_connections 2023-10-30T20:24:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7fb993ade640 1 --2- 172.21.15.69:0/1914704085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb98c13eb80 0x7fb98c13efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7fb993ade640 1 --2- 172.21.15.69:0/1914704085 >> v2:172.21.15.69:3300/0 conn(0x7fb98c145360 0x7fb98c1457f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7fb993ade640 1 --2- 172.21.15.69:0/1914704085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98c0aca50 0x7fb98c06db60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7fb993ade640 1 -- 172.21.15.69:0/1914704085 >> 172.21.15.69:0/1914704085 conn(0x7fb98c0afba0 msgr2=0x7fb98c0affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7fb993ade640 1 -- 172.21.15.69:0/1914704085 shutdown_connections 2023-10-30T20:24:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7fb993ade640 1 -- 172.21.15.69:0/1914704085 wait complete. 2023-10-30T20:24:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7f0f7dfca640 1 --2- 172.21.15.69:0/2087719441 >> v2:172.21.15.183:6804/1046221745 conn(0x7f0f5c0725d0 0x7f0f5c0749f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.198 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7f0f6dffb640 10 client.4279.objecter ms_handle_connect 0x7f0f5c0725d0 2023-10-30T20:24:07.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7fb993ade640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.001+0000 7fb993ade640 10 monclient: monmap: 2023-10-30T20:24:07.199 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.199 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.200 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.200 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.200 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.200 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.201 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.201 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.201 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.201 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.002+0000 7fb993ade640 1 librados: starting msgr at 2023-10-30T20:24:07.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.002+0000 7fb993ade640 1 librados: starting objecter 2023-10-30T20:24:07.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.002+0000 7fb993ade640 1 Processor -- start 2023-10-30T20:24:07.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.002+0000 7fb993ade640 1 -- start start 2023-10-30T20:24:07.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.002+0000 7fb993ade640 1 librados: setting wanted keys 2023-10-30T20:24:07.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.002+0000 7fb993ade640 1 librados: calling monclient init 2023-10-30T20:24:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.002+0000 7fb993ade640 10 monclient: init 2023-10-30T20:24:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.002+0000 7fb993ade640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.002+0000 7fb993ade640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T20:24:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.002+0000 7fb993ade640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb98c0aca50 0x7fb98c1d9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7fb993ade640 10 monclient(hunting): picked mon.noname-c con 0x7fb98c0aca50 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7fb993ade640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98c13eb80 0x7fb98c139f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7fb993ade640 10 monclient(hunting): picked mon.noname-b con 0x7fb98c13eb80 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7fb993ade640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb98c145360 0x7fb98c13a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7fb993ade640 10 monclient(hunting): picked mon.noname-a con 0x7fb98c145360 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7fb993ade640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7fb993ade640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c0aec60 con 0x7fb98c145360 2023-10-30T20:24:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7fb993ade640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7fb993ade640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c11b0b0 con 0x7fb98c13eb80 2023-10-30T20:24:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7fb993ade640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7fb993ade640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c13aa50 con 0x7fb98c0aca50 2023-10-30T20:24:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7fb993ade640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7fb993ade640 10 monclient(hunting): authenticate will time out at 1570.291916s 2023-10-30T20:24:07.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 1 -- 172.21.15.69:0/3287870425 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f1ce4058c60 con 0x7f1cec0070e0 2023-10-30T20:24:07.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 10 client.4284.objecter ms_dispatch 0x7f1cf4000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:07.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 3 client.4284.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:07.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 3 client.4284.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:07.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 10 client.4284.objecter checking command 1 2023-10-30T20:24:07.208 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 1 --2- 172.21.15.69:0/3287870425 >> v2:172.21.15.69:6809/1824616966 conn(0x7f1ce0072620 0x7f1ce0074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.208 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 20 client.4284.objecter _get_session s=0x7f1ce4012630 osd=3 3 2023-10-30T20:24:07.208 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 20 client.4284.objecter put_session s=0x7f1ce4012630 osd=3 3 2023-10-30T20:24:07.208 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 15 client.4284.objecter _session_command_op_remove -1 1 2023-10-30T20:24:07.208 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 20 client.4284.objecter _get_session s=0x7f1ce4012630 osd=3 3 2023-10-30T20:24:07.208 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 20 client.4284.objecter get_session s=0x7f1ce4012630 osd=3 3 2023-10-30T20:24:07.209 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 15 client.4284.objecter _session_command_op_assign 3 1 2023-10-30T20:24:07.209 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 20 client.4284.objecter put_session s=0x7f1ce4012630 osd=3 4 2023-10-30T20:24:07.209 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 10 client.4284.objecter _send_command 1 2023-10-30T20:24:07.209 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 1 -- 172.21.15.69:0/3287870425 --> v2:172.21.15.69:6809/1824616966 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ce40052d0 con 0x7f1ce0072620 2023-10-30T20:24:07.209 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 20 client.4284.objecter dump_active .. 0 homeless 2023-10-30T20:24:07.209 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 1 -- 172.21.15.69:0/3287870425 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ce40546d0 con 0x7f1cec0070e0 2023-10-30T20:24:07.210 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.003+0000 7f1cdbfff640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T20:24:07.210 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.004+0000 7f1cf37fe640 1 --2- 172.21.15.69:0/3287870425 >> v2:172.21.15.69:6809/1824616966 conn(0x7f1ce0072620 0x7f1ce0074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.210 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.004+0000 7f1cf37fe640 10 monclient: get_auth_request con 0x7f1ce0072620 auth_method 0 2023-10-30T20:24:07.211 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae91ffb640 1 -- 172.21.15.69:0/194359510 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae880317c0 con 0x7fae940b13c0 2023-10-30T20:24:07.212 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.212 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae91ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:07.212 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae91ffb640 10 monclient: dump: 2023-10-30T20:24:07.212 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.212 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.213 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.213 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.213 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.213 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.213 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.213 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.214 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.214 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.214 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae91ffb640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.214 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae91ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.214 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.214 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae91ffb640 1 -- 172.21.15.69:0/194359510 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8802e070 con 0x7fae940b13c0 2023-10-30T20:24:07.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae91ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae91ffb640 1 -- 172.21.15.69:0/194359510 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae88031b50 con 0x7fae940b13c0 2023-10-30T20:24:07.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae91ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:07.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae91ffb640 10 monclient: dump: 2023-10-30T20:24:07.216 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.216 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.216 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.216 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.216 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.216 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.216 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.217 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.217 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.217 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.217 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae99621640 5 monclient: authenticate success, global_id 4287 2023-10-30T20:24:07.217 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae99621640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:07.217 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.005+0000 7fae99621640 4 set_mon_vals no callback set 2023-10-30T20:24:07.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.006+0000 7fae99621640 10 monclient: shutdown 2023-10-30T20:24:07.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.006+0000 7fae99621640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.006+0000 7fae99621640 1 -- 172.21.15.69:0/194359510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae940b13c0 msgr2=0x7fae94141eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.006+0000 7fae99621640 1 --2- 172.21.15.69:0/194359510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae940b13c0 0x7fae94141eb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fae88009870 tx=0x7fae8802b860 comp rx=0 tx=0).stop 2023-10-30T20:24:07.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.006+0000 7fae99621640 1 -- 172.21.15.69:0/194359510 shutdown_connections 2023-10-30T20:24:07.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.006+0000 7fae99621640 1 --2- 172.21.15.69:0/194359510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae940b13c0 0x7fae94141eb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.006+0000 7fae99621640 1 --2- 172.21.15.69:0/194359510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae940a8e40 0x7fae9413efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.006+0000 7fae99621640 1 -- 172.21.15.69:0/194359510 >> 172.21.15.69:0/194359510 conn(0x7fae940b7180 msgr2=0x7fae940b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.006+0000 7fae99621640 1 -- 172.21.15.69:0/194359510 shutdown_connections 2023-10-30T20:24:07.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.006+0000 7fae99621640 1 -- 172.21.15.69:0/194359510 wait complete. 2023-10-30T20:24:07.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.007+0000 7fae99621640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.007+0000 7fae99621640 10 monclient: monmap: 2023-10-30T20:24:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.221 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.221 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.221 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.221 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.221 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.007+0000 7fae99621640 1 librados: starting msgr at 2023-10-30T20:24:07.222 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.007+0000 7fae99621640 1 librados: starting objecter 2023-10-30T20:24:07.222 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.007+0000 7fae99621640 1 Processor -- start 2023-10-30T20:24:07.222 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.007+0000 7fae99621640 1 -- start start 2023-10-30T20:24:07.222 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.007+0000 7fae99621640 1 librados: setting wanted keys 2023-10-30T20:24:07.222 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.007+0000 7fae99621640 1 librados: calling monclient init 2023-10-30T20:24:07.222 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.007+0000 7fae99621640 10 monclient: init 2023-10-30T20:24:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:24:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae940a8e40 0x7fae940b2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 10 monclient(hunting): picked mon.noname-b con 0x7fae940a8e40 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae940ad9f0 0x7fae941d9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 10 monclient(hunting): picked mon.noname-c con 0x7fae940ad9f0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae941d7190 0x7fae940ac770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 10 monclient(hunting): picked mon.noname-a con 0x7fae941d7190 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae940b7e80 con 0x7fae941d7190 2023-10-30T20:24:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae941c9ae0 con 0x7fae940a8e40 2023-10-30T20:24:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae941da310 con 0x7fae940ad9f0 2023-10-30T20:24:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7fae99621640 10 monclient(hunting): authenticate will time out at 1570.297485s 2023-10-30T20:24:07.226 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7f1cf37fe640 1 --2- 172.21.15.69:0/3287870425 >> v2:172.21.15.69:6809/1824616966 conn(0x7f1ce0072620 0x7f1ce0074a40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.226 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.008+0000 7f1cdbfff640 10 client.4284.objecter ms_handle_connect 0x7f1ce0072620 2023-10-30T20:24:07.227 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f59210b7640 1 -- 172.21.15.69:0/2027567513 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591402cb40 con 0x7f591c1425c0 2023-10-30T20:24:07.227 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f59210b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.227 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f59210b7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:07.228 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f59210b7640 10 monclient: dump: 2023-10-30T20:24:07.228 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.228 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.228 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.228 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.228 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.229 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.229 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.229 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.229 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.229 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.229 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f59210b7640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.230 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f59210b7640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.230 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f59210b7640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.230 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f59210b7640 1 -- 172.21.15.69:0/2027567513 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591402ccb0 con 0x7f591c1425c0 2023-10-30T20:24:07.230 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f59210b7640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.230 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f59210b7640 1 -- 172.21.15.69:0/2027567513 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914034660 con 0x7f591c1425c0 2023-10-30T20:24:07.230 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f592431d640 5 monclient: authenticate success, global_id 4290 2023-10-30T20:24:07.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f592431d640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:07.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f592431d640 4 set_mon_vals no callback set 2023-10-30T20:24:07.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f59210b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f59210b7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:07.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f59210b7640 10 monclient: dump: 2023-10-30T20:24:07.231 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.232 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.232 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.232 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.232 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.232 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.232 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.233 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.233 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.233 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.233 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f592431d640 10 monclient: shutdown 2023-10-30T20:24:07.233 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f592431d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.233 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f592431d640 1 -- 172.21.15.69:0/2027567513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f591c1425c0 msgr2=0x7f591c14d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.234 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.009+0000 7f592431d640 1 --2- 172.21.15.69:0/2027567513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f591c1425c0 0x7f591c14d330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5914009d90 tx=0x7f591402bd90 comp rx=0 tx=0).stop 2023-10-30T20:24:07.234 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feeb67fc640 1 -- 172.21.15.69:0/689861333 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea8031740 con 0x7feeb80a8e40 2023-10-30T20:24:07.234 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feeb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.234 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feeb67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:07.235 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feeb67fc640 10 monclient: dump: 2023-10-30T20:24:07.235 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.235 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.235 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.235 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.235 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.236 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.236 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.236 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.236 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.236 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feeb67fc640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feeb67fc640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.237 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feeb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.237 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feeb67fc640 1 -- 172.21.15.69:0/689861333 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea80318b0 con 0x7feeb80a8e40 2023-10-30T20:24:07.237 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feeb67fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.237 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feeb67fc640 1 -- 172.21.15.69:0/689861333 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea8031c50 con 0x7feeb80a8e40 2023-10-30T20:24:07.237 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feeb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.237 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feeb67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.238 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feeb67fc640 10 monclient: dump: 2023-10-30T20:24:07.238 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.238 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.238 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.238 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.239 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.239 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.239 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.239 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.239 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.239 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.240 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7f592431d640 1 -- 172.21.15.69:0/2027567513 shutdown_connections 2023-10-30T20:24:07.240 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7f592431d640 1 --2- 172.21.15.69:0/2027567513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f591c1425c0 0x7f591c14d330 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.240 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7f592431d640 1 -- 172.21.15.69:0/2027567513 >> 172.21.15.69:0/2027567513 conn(0x7f591c0b7180 msgr2=0x7f591c0b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.240 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7f592431d640 1 -- 172.21.15.69:0/2027567513 shutdown_connections 2023-10-30T20:24:07.240 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7f592431d640 1 -- 172.21.15.69:0/2027567513 wait complete. 2023-10-30T20:24:07.241 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feebe11d640 5 monclient: authenticate success, global_id 4282 2023-10-30T20:24:07.241 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feebe11d640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:07.241 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.010+0000 7feebe11d640 4 set_mon_vals no callback set 2023-10-30T20:24:07.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7f592431d640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7f592431d640 10 monclient: monmap: 2023-10-30T20:24:07.242 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.242 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.242 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.242 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.243 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.243 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.243 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.243 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.243 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.244 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.244 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7feebe11d640 10 monclient: shutdown 2023-10-30T20:24:07.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7feebe11d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7feebe11d640 1 -- 172.21.15.69:0/689861333 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feeb80a8e40 msgr2=0x7feeb813efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7feebe11d640 1 --2- 172.21.15.69:0/689861333 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feeb80a8e40 0x7feeb813efc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feeb80b8030 tx=0x7feea802b7e0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7fb991079640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98c13eb80 0x7fb98c139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7feebe11d640 1 -- 172.21.15.69:0/689861333 shutdown_connections 2023-10-30T20:24:07.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7feebe11d640 1 --2- 172.21.15.69:0/689861333 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feeb813f500 0x7feeb81423f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7feebe11d640 1 --2- 172.21.15.69:0/689861333 >> v2:172.21.15.69:3300/0 conn(0x7feeb80b13c0 0x7feeb8141eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7f592431d640 1 librados: starting msgr at 2023-10-30T20:24:07.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7feebe11d640 1 --2- 172.21.15.69:0/689861333 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feeb80a8e40 0x7feeb813efc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7f592431d640 1 librados: starting objecter 2023-10-30T20:24:07.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7feebe11d640 1 -- 172.21.15.69:0/689861333 >> 172.21.15.69:0/689861333 conn(0x7feeb80b7180 msgr2=0x7feeb80b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7feebe11d640 1 -- 172.21.15.69:0/689861333 shutdown_connections 2023-10-30T20:24:07.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7feebe11d640 1 -- 172.21.15.69:0/689861333 wait complete. 2023-10-30T20:24:07.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7f592431d640 1 Processor -- start 2023-10-30T20:24:07.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7f592431d640 1 -- start start 2023-10-30T20:24:07.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7f592431d640 1 librados: setting wanted keys 2023-10-30T20:24:07.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7f592431d640 1 librados: calling monclient init 2023-10-30T20:24:07.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.011+0000 7f592431d640 10 monclient: init 2023-10-30T20:24:07.251 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.012+0000 7feebe11d640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.251 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.012+0000 7feebe11d640 10 monclient: monmap: 2023-10-30T20:24:07.251 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.251 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.251 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.252 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.252 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.252 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.252 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.252 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.253 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.253 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.253 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.012+0000 7fb991079640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98c13eb80 0x7fb98c139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36494/0 (socket says 172.21.15.69:36494) 2023-10-30T20:24:07.253 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.012+0000 7fb991079640 1 -- 172.21.15.69:0/1022410054 learned_addr learned my addr 172.21.15.69:0/1022410054 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.253 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.012+0000 7fb991079640 10 monclient(hunting): get_auth_request con 0x7fb98c13eb80 auth_method 0 2023-10-30T20:24:07.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.012+0000 7fb991079640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.012+0000 7fb991079640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.012+0000 7fb991079640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.012+0000 7f592431d640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.012+0000 7f592431d640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:24:07.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.012+0000 7f592431d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f591c0a8e40 0x7f591c11f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7f592431d640 10 monclient(hunting): picked mon.noname-a con 0x7f591c0a8e40 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7f592431d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f591c1425c0 0x7f591c1457a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7f592431d640 10 monclient(hunting): picked mon.noname-b con 0x7f591c1425c0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7f592431d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f591c13d360 0x7f591c13d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7feebe11d640 1 librados: starting msgr at 2023-10-30T20:24:07.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7feebe11d640 1 librados: starting objecter 2023-10-30T20:24:07.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7feebe11d640 1 Processor -- start 2023-10-30T20:24:07.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7f592431d640 10 monclient(hunting): picked mon.noname-c con 0x7f591c13d360 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7fb99207b640 1 --2- 172.21.15.69:0/1022410054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb98c145360 0x7fb98c13a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7f592431d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7f592431d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0a9930 con 0x7f591c0a8e40 2023-10-30T20:24:07.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7f592431d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7f592431d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c142bb0 con 0x7f591c1425c0 2023-10-30T20:24:07.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7f592431d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7f592431d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591c11f880 con 0x7f591c13d360 2023-10-30T20:24:07.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7f592431d640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7f592431d640 10 monclient(hunting): authenticate will time out at 1570.302484s 2023-10-30T20:24:07.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7fb991079640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7fb991079640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7fb991079640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7feebe11d640 1 -- start start 2023-10-30T20:24:07.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.013+0000 7feebe11d640 1 librados: setting wanted keys 2023-10-30T20:24:07.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7feebe11d640 1 librados: calling monclient init 2023-10-30T20:24:07.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7feebe11d640 10 monclient: init 2023-10-30T20:24:07.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7f59220b9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f591c0a8e40 0x7f591c11f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7f59220b9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f591c0a8e40 0x7f591c11f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45290/0 (socket says 172.21.15.69:45290) 2023-10-30T20:24:07.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7f59220b9640 1 -- 172.21.15.69:0/3480831939 learned_addr learned my addr 172.21.15.69:0/3480831939 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7f59220b9640 10 monclient(hunting): get_auth_request con 0x7f591c0a8e40 auth_method 0 2023-10-30T20:24:07.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7f59220b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7f59220b9640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7fb99207b640 10 monclient(hunting): get_auth_request con 0x7fb98c145360 auth_method 0 2023-10-30T20:24:07.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7fb99207b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7fb99207b640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7fb99207b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7f59220b9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7feebe11d640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.014+0000 7feebe11d640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T20:24:07.266 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7feebe11d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feeb80a8e40 0x7feeb80ac070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.266 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7f9beeffd640 1 -- 172.21.15.69:0/1950889256 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f9bf40073e0 con 0x7f9bf0030b10 2023-10-30T20:24:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7feebe11d640 10 monclient(hunting): picked mon.noname-c con 0x7feeb80a8e40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7feebe11d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feeb80ab6b0 0x7feeb813b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7feebe11d640 10 monclient(hunting): picked mon.noname-b con 0x7feeb80ab6b0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7feebe11d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feeb813b760 0x7feeb81da390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.268 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7f59220b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.268 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7f59220b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.268 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7f59220b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7feebe11d640 10 monclient(hunting): picked mon.noname-a con 0x7feeb813b760 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7feebe11d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7feebe11d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb8142570 con 0x7feeb813b760 2023-10-30T20:24:07.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7feebe11d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7feebe11d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb80abc60 con 0x7feeb80ab6b0 2023-10-30T20:24:07.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7feebe11d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7feebe11d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb8145ae0 con 0x7feeb80a8e40 2023-10-30T20:24:07.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7feebe11d640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7feebe11d640 10 monclient(hunting): authenticate will time out at 1570.304266s 2023-10-30T20:24:07.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7f9beeffd640 1 --2- 172.21.15.69:0/1950889256 >> v2:172.21.15.69:6800/1876307353 conn(0x7f9bdc06c810 0x7f9bdc06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7fb99207b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7fb99207b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7fb99207b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7f9bfe2dd640 1 --2- 172.21.15.69:0/1950889256 >> v2:172.21.15.69:6800/1876307353 conn(0x7f9bdc06c810 0x7f9bdc06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.015+0000 7f9bfe2dd640 10 monclient: get_auth_request con 0x7f9bdc06c810 auth_method 0 2023-10-30T20:24:07.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.016+0000 7f9bfe2dd640 1 --2- 172.21.15.69:0/1950889256 >> v2:172.21.15.69:6800/1876307353 conn(0x7f9bdc06c810 0x7f9bdc06ecd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9be8036000 tx=0x7f9be8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.016+0000 7f9beeffd640 10 client.4271.objecter ms_handle_connect 0x7f9bdc06c810 2023-10-30T20:24:07.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.018+0000 7fb991079640 10 monclient(hunting): handle_auth_done global_id 4294 payload 824 2023-10-30T20:24:07.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.018+0000 7fb991079640 1 -- 172.21.15.69:0/1022410054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb98c0aca50 msgr2=0x7fb98c1d9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.018+0000 7fb991079640 1 --2- 172.21.15.69:0/1022410054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb98c0aca50 0x7fb98c1d9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.018+0000 7fb991079640 1 -- 172.21.15.69:0/1022410054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb98c145360 msgr2=0x7fb98c13a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.018+0000 7fb991079640 1 --2- 172.21.15.69:0/1022410054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb98c145360 0x7fb98c13a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.018+0000 7fb991079640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.018+0000 7fb991079640 1 monclient: found mon.noname-b 2023-10-30T20:24:07.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.018+0000 7fb991079640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.018+0000 7fb991079640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.018+0000 7fb991079640 1 -- 172.21.15.69:0/1022410054 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98c11acd0 con 0x7fb98c13eb80 2023-10-30T20:24:07.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.018+0000 7f59218b8640 1 --2- 172.21.15.69:0/3480831939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f591c1425c0 0x7f591c1457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.019+0000 7f59218b8640 10 monclient(hunting): get_auth_request con 0x7f591c1425c0 auth_method 0 2023-10-30T20:24:07.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.019+0000 7f59218b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.019+0000 7f59218b8640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.019+0000 7f59218b8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 1 -- 172.21.15.69:0/1950889256 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f9bf40579e0 con 0x7f9bf0030b10 2023-10-30T20:24:07.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 10 client.4271.objecter ms_dispatch 0x7f9bf8000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:07.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 3 client.4271.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:07.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 3 client.4271.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:07.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 10 client.4271.objecter checking command 1 2023-10-30T20:24:07.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 1 --2- 172.21.15.69:0/1950889256 >> v2:172.21.15.183:6808/1361328034 conn(0x7f9bdc072620 0x7f9bdc074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 20 client.4271.objecter _get_session s=0x7f9bf4007270 osd=7 3 2023-10-30T20:24:07.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 20 client.4271.objecter put_session s=0x7f9bf4007270 osd=7 3 2023-10-30T20:24:07.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 15 client.4271.objecter _session_command_op_remove -1 1 2023-10-30T20:24:07.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 20 client.4271.objecter _get_session s=0x7f9bf4007270 osd=7 3 2023-10-30T20:24:07.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 20 client.4271.objecter get_session s=0x7f9bf4007270 osd=7 3 2023-10-30T20:24:07.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 15 client.4271.objecter _session_command_op_assign 7 1 2023-10-30T20:24:07.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 20 client.4271.objecter put_session s=0x7f9bf4007270 osd=7 4 2023-10-30T20:24:07.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 10 client.4271.objecter _send_command 1 2023-10-30T20:24:07.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 1 -- 172.21.15.69:0/1950889256 --> v2:172.21.15.183:6808/1361328034 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9bf4005ce0 con 0x7f9bdc072620 2023-10-30T20:24:07.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 20 client.4271.objecter dump_active .. 0 homeless 2023-10-30T20:24:07.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 1 -- 172.21.15.69:0/1950889256 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9bf4053d20 con 0x7f9bf0030b10 2023-10-30T20:24:07.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.020+0000 7f9beeffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T20:24:07.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.021+0000 7f9bfeade640 1 --2- 172.21.15.69:0/1950889256 >> v2:172.21.15.183:6808/1361328034 conn(0x7f9bdc072620 0x7f9bdc074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.021+0000 7f9bfeade640 10 monclient: get_auth_request con 0x7f9bdc072620 auth_method 0 2023-10-30T20:24:07.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.021+0000 7fae927fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae940ad9f0 0x7fae941d9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.021+0000 7f59220b9640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2023-10-30T20:24:07.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.021+0000 7f59220b9640 1 -- 172.21.15.69:0/3480831939 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f591c13d360 msgr2=0x7f591c13d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.021+0000 7f59220b9640 1 --2- 172.21.15.69:0/3480831939 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f591c13d360 0x7f591c13d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.021+0000 7f59220b9640 1 -- 172.21.15.69:0/3480831939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f591c1425c0 msgr2=0x7f591c1457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.021+0000 7f59220b9640 1 --2- 172.21.15.69:0/3480831939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f591c1425c0 0x7f591c1457a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.021+0000 7f59220b9640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.021+0000 7f59220b9640 1 monclient: found mon.noname-a 2023-10-30T20:24:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.021+0000 7f59220b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.021+0000 7f59220b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.021+0000 7f59220b9640 1 -- 172.21.15.69:0/3480831939 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c1448f0 con 0x7f591c0a8e40 2023-10-30T20:24:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.021+0000 7f59218b8640 1 --2- 172.21.15.69:0/3480831939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f591c1425c0 0x7f591c1457a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T20:24:07.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.023+0000 7fb991079640 1 --2- 172.21.15.69:0/1022410054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98c13eb80 0x7fb98c139f60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00d800 tx=0x7fb97c00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.023+0000 7fb9827fc640 10 client.?.objecter ms_handle_connect 0x7fb98c13eb80 2023-10-30T20:24:07.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.023+0000 7fb9827fc640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:07.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.024+0000 7fae92ffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae940a8e40 0x7fae940b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.024+0000 7f9bfeade640 1 --2- 172.21.15.69:0/1950889256 >> v2:172.21.15.183:6808/1361328034 conn(0x7f9bdc072620 0x7f9bdc074a40 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.024+0000 7f9beeffd640 10 client.4271.objecter ms_handle_connect 0x7f9bdc072620 2023-10-30T20:24:07.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.026+0000 7fae92ffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae940a8e40 0x7fae940b2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36500/0 (socket says 172.21.15.69:36500) 2023-10-30T20:24:07.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.026+0000 7fae92ffd640 1 -- 172.21.15.69:0/1377797277 learned_addr learned my addr 172.21.15.69:0/1377797277 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.026+0000 7fae92ffd640 10 monclient(hunting): get_auth_request con 0x7fae940a8e40 auth_method 0 2023-10-30T20:24:07.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.026+0000 7fae92ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.026+0000 7fae92ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.026+0000 7fae92ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.027+0000 7feeb6ffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feeb80ab6b0 0x7feeb813b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.027+0000 7fae937fe640 1 --2- 172.21.15.69:0/1377797277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae941d7190 0x7fae940ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.027+0000 7fae92ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.027+0000 7fae92ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.027+0000 7fae92ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.027+0000 7feeb6ffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feeb80ab6b0 0x7feeb813b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36512/0 (socket says 172.21.15.69:36512) 2023-10-30T20:24:07.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.027+0000 7feeb6ffd640 1 -- 172.21.15.69:0/553467047 learned_addr learned my addr 172.21.15.69:0/553467047 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.027+0000 7feeb6ffd640 10 monclient(hunting): get_auth_request con 0x7feeb80ab6b0 auth_method 0 2023-10-30T20:24:07.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.027+0000 7feeb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.027+0000 7feeb6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.027+0000 7feeb6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.029+0000 7fae937fe640 10 monclient(hunting): get_auth_request con 0x7fae941d7190 auth_method 0 2023-10-30T20:24:07.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.029+0000 7fae937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.029+0000 7fae937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.029+0000 7fae937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.029+0000 7f59220b9640 1 --2- 172.21.15.69:0/3480831939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f591c0a8e40 0x7f591c11f280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f591800c470 tx=0x7f591800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.029+0000 7f5912ffd640 10 client.?.objecter ms_handle_connect 0x7f591c0a8e40 2023-10-30T20:24:07.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.029+0000 7f5912ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:07.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.030+0000 7fae937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.030+0000 7fae937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.030+0000 7fae937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.030+0000 7feeb7fff640 1 --2- 172.21.15.69:0/553467047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feeb813b760 0x7feeb81da390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.030+0000 7feeb7fff640 10 monclient(hunting): get_auth_request con 0x7feeb813b760 auth_method 0 2023-10-30T20:24:07.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.030+0000 7feeb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.030+0000 7feeb7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.030+0000 7feeb7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.032+0000 7feeb6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.032+0000 7feeb6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.033+0000 7feeb6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.033+0000 7fae92ffd640 10 monclient(hunting): handle_auth_done global_id 4300 payload 824 2023-10-30T20:24:07.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.033+0000 7fae92ffd640 1 -- 172.21.15.69:0/1377797277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae940ad9f0 msgr2=0x7fae941d9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.033+0000 7fae92ffd640 1 --2- 172.21.15.69:0/1377797277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae940ad9f0 0x7fae941d9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.033+0000 7fae92ffd640 1 -- 172.21.15.69:0/1377797277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae941d7190 msgr2=0x7fae940ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.033+0000 7fae92ffd640 1 --2- 172.21.15.69:0/1377797277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae941d7190 0x7fae940ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.033+0000 7fae92ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.033+0000 7fae92ffd640 1 monclient: found mon.noname-b 2023-10-30T20:24:07.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.033+0000 7fae92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.033+0000 7fae92ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.033+0000 7fae92ffd640 1 -- 172.21.15.69:0/1377797277 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae941d6c70 con 0x7fae940a8e40 2023-10-30T20:24:07.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.036+0000 7feeb6ffd640 10 monclient(hunting): handle_auth_done global_id 4303 payload 824 2023-10-30T20:24:07.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.036+0000 7feeb6ffd640 1 -- 172.21.15.69:0/553467047 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feeb80a8e40 msgr2=0x7feeb80ac070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.036+0000 7feeb6ffd640 1 --2- 172.21.15.69:0/553467047 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feeb80a8e40 0x7feeb80ac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.036+0000 7feeb6ffd640 1 -- 172.21.15.69:0/553467047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feeb813b760 msgr2=0x7feeb81da390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.036+0000 7feeb6ffd640 1 --2- 172.21.15.69:0/553467047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feeb813b760 0x7feeb81da390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.036+0000 7feeb6ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.036+0000 7feeb6ffd640 1 monclient: found mon.noname-b 2023-10-30T20:24:07.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.036+0000 7feeb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.036+0000 7feeb6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.036+0000 7feeb6ffd640 1 -- 172.21.15.69:0/553467047 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb813bbe0 con 0x7feeb80ab6b0 2023-10-30T20:24:07.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.039+0000 7fae92ffd640 1 --2- 172.21.15.69:0/1377797277 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae940a8e40 0x7fae940b2340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fae8400b4d0 tx=0x7fae8400b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.039+0000 7fae7b7fe640 10 client.?.objecter ms_handle_connect 0x7fae940a8e40 2023-10-30T20:24:07.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.039+0000 7fae7b7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:07.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.040+0000 7feeb6ffd640 1 --2- 172.21.15.69:0/553467047 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feeb80ab6b0 0x7feeb813b220 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feea400b480 tx=0x7feea400b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.040+0000 7feeaffff640 10 client.?.objecter ms_handle_connect 0x7feeb80ab6b0 2023-10-30T20:24:07.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.040+0000 7feeaffff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:07.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f5912ffd640 1 -- 172.21.15.69:0/3480831939 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59180035f0 con 0x7f591c0a8e40 2023-10-30T20:24:07.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f5912ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f5912ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:07.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f5912ffd640 10 monclient: dump: 2023-10-30T20:24:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.304 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.304 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.304 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.304 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f5912ffd640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f5912ffd640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f5912ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f5912ffd640 1 -- 172.21.15.69:0/3480831939 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591800e040 con 0x7f591c0a8e40 2023-10-30T20:24:07.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f5912ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f5912ffd640 1 -- 172.21.15.69:0/3480831939 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59180039c0 con 0x7f591c0a8e40 2023-10-30T20:24:07.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f5912ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f5912ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:07.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f5912ffd640 10 monclient: dump: 2023-10-30T20:24:07.306 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.306 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.306 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.306 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.307 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.307 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.307 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.307 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.307 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.307 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f592431d640 5 monclient: authenticate success, global_id 4293 2023-10-30T20:24:07.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f59208b6640 4 set_mon_vals no callback set 2023-10-30T20:24:07.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f592431d640 10 monclient: _renew_subs 2023-10-30T20:24:07.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f592431d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f592431d640 1 -- 172.21.15.69:0/3480831939 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591c13dd50 con 0x7f591c0a8e40 2023-10-30T20:24:07.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f592431d640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f592431d640 10 monclient: _renew_subs 2023-10-30T20:24:07.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f592431d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f592431d640 1 -- 172.21.15.69:0/3480831939 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f591c145ec0 con 0x7f591c0a8e40 2023-10-30T20:24:07.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.041+0000 7f592431d640 1 librados: init done 2023-10-30T20:24:07.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.042+0000 7f4620ff9640 1 -- 172.21.15.69:0/3841363599 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f461804d3f0 con 0x7f4624071710 2023-10-30T20:24:07.310 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.042+0000 7f592431d640 10 client.4293.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:07.310 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.042+0000 7f592431d640 15 client.4293.objecter _session_command_op_assign -1 1 2023-10-30T20:24:07.310 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.042+0000 7f592431d640 20 client.4293.objecter _get_session osd=-1 returning homeless 2023-10-30T20:24:07.310 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.042+0000 7f592431d640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.042+0000 7f592431d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.042+0000 7f592431d640 1 -- 172.21.15.69:0/3480831939 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f591c0a97b0 con 0x7f591c0a8e40 2023-10-30T20:24:07.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.043+0000 7f4620ff9640 1 --2- 172.21.15.69:0/3841363599 >> v2:172.21.15.69:6800/1876307353 conn(0x7f461006c7c0 0x7f461006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.043+0000 7f46237fe640 1 --2- 172.21.15.69:0/3841363599 >> v2:172.21.15.69:6800/1876307353 conn(0x7f461006c7c0 0x7f461006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.043+0000 7f46237fe640 10 monclient: get_auth_request con 0x7f461006c7c0 auth_method 0 2023-10-30T20:24:07.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.043+0000 7f46237fe640 1 --2- 172.21.15.69:0/3841363599 >> v2:172.21.15.69:6800/1876307353 conn(0x7f461006c7c0 0x7f461006ec80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f46241478a0 tx=0x7f4614008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.043+0000 7f4620ff9640 10 client.4274.objecter ms_handle_connect 0x7f461006c7c0 2023-10-30T20:24:07.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feeaffff640 1 -- 172.21.15.69:0/553467047 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4010870 con 0x7feeb80ab6b0 2023-10-30T20:24:07.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feeaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feeaffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:07.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feeaffff640 10 monclient: dump: 2023-10-30T20:24:07.315 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.315 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.315 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.316 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.316 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.316 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.316 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.316 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.316 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.317 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feeaffff640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feeaffff640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feeaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feeaffff640 1 -- 172.21.15.69:0/553467047 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea40109e0 con 0x7feeb80ab6b0 2023-10-30T20:24:07.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feebe11d640 5 monclient: authenticate success, global_id 4303 2023-10-30T20:24:07.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feebe11d640 10 monclient: _renew_subs 2023-10-30T20:24:07.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feebe11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feebe11d640 1 -- 172.21.15.69:0/553467047 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb813bf30 con 0x7feeb80ab6b0 2023-10-30T20:24:07.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feebe11d640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feeaffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feeaffff640 1 -- 172.21.15.69:0/553467047 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4010da0 con 0x7feeb80ab6b0 2023-10-30T20:24:07.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feeaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feeb5ffb640 4 set_mon_vals no callback set 2023-10-30T20:24:07.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feeaffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feeaffff640 10 monclient: dump: 2023-10-30T20:24:07.319 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.319 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.320 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.320 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.320 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.320 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.320 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.320 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.321 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.321 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feebe11d640 10 monclient: _renew_subs 2023-10-30T20:24:07.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feebe11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feebe11d640 1 -- 172.21.15.69:0/553467047 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb81daab0 con 0x7feeb80ab6b0 2023-10-30T20:24:07.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.047+0000 7feebe11d640 1 librados: init done 2023-10-30T20:24:07.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb9827fc640 1 -- 172.21.15.69:0/1022410054 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c0096a0 con 0x7fb98c13eb80 2023-10-30T20:24:07.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb9827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb9827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:07.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb9827fc640 10 monclient: dump: 2023-10-30T20:24:07.323 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.323 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.323 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.323 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.324 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.324 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.324 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.324 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.324 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.324 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb9827fc640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb9827fc640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb9827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb9827fc640 1 -- 172.21.15.69:0/1022410054 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c009810 con 0x7fb98c13eb80 2023-10-30T20:24:07.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb9827fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7feebe11d640 10 client.4303.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:07.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7feebe11d640 15 client.4303.objecter _session_command_op_assign -1 1 2023-10-30T20:24:07.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7feebe11d640 20 client.4303.objecter _get_session osd=-1 returning homeless 2023-10-30T20:24:07.327 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7feebe11d640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.327 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7feebe11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.327 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7feebe11d640 1 -- 172.21.15.69:0/553467047 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feeb80a97b0 con 0x7feeb80ab6b0 2023-10-30T20:24:07.328 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 1 -- 172.21.15.69:0/3841363599 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f4618079e70 con 0x7f4624071710 2023-10-30T20:24:07.328 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 10 client.4274.objecter ms_dispatch 0x7f4624000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:07.328 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 3 client.4274.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:07.328 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 3 client.4274.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:07.328 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 10 client.4274.objecter checking command 1 2023-10-30T20:24:07.328 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 1 --2- 172.21.15.69:0/3841363599 >> v2:172.21.15.69:6805/2300065874 conn(0x7f46100725d0 0x7f46100749f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 20 client.4274.objecter _get_session s=0x7f4618032c30 osd=2 3 2023-10-30T20:24:07.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 20 client.4274.objecter put_session s=0x7f4618032c30 osd=2 3 2023-10-30T20:24:07.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 15 client.4274.objecter _session_command_op_remove -1 1 2023-10-30T20:24:07.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 20 client.4274.objecter _get_session s=0x7f4618032c30 osd=2 3 2023-10-30T20:24:07.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 20 client.4274.objecter get_session s=0x7f4618032c30 osd=2 3 2023-10-30T20:24:07.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 15 client.4274.objecter _session_command_op_assign 2 1 2023-10-30T20:24:07.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 20 client.4274.objecter put_session s=0x7f4618032c30 osd=2 4 2023-10-30T20:24:07.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 10 client.4274.objecter _send_command 1 2023-10-30T20:24:07.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 1 -- 172.21.15.69:0/3841363599 --> v2:172.21.15.69:6805/2300065874 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4618032860 con 0x7f46100725d0 2023-10-30T20:24:07.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 20 client.4274.objecter dump_active .. 0 homeless 2023-10-30T20:24:07.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 1 -- 172.21.15.69:0/3841363599 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46180094d0 con 0x7f4624071710 2023-10-30T20:24:07.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7f4620ff9640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T20:24:07.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb9827fc640 1 -- 172.21.15.69:0/1022410054 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c0096a0 con 0x7fb98c13eb80 2023-10-30T20:24:07.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb9827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb9827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb983fff640 4 set_mon_vals no callback set 2023-10-30T20:24:07.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb9827fc640 10 monclient: dump: 2023-10-30T20:24:07.332 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.332 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.333 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.333 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.333 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.333 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.333 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.333 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.334 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.334 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb993ade640 5 monclient: authenticate success, global_id 4294 2023-10-30T20:24:07.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb993ade640 10 monclient: _renew_subs 2023-10-30T20:24:07.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb993ade640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb993ade640 1 -- 172.21.15.69:0/1022410054 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98c1da9d0 con 0x7fb98c13eb80 2023-10-30T20:24:07.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb993ade640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb993ade640 10 monclient: _renew_subs 2023-10-30T20:24:07.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb993ade640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb993ade640 1 -- 172.21.15.69:0/1022410054 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb98c11b5d0 con 0x7fb98c13eb80 2023-10-30T20:24:07.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.048+0000 7fb993ade640 1 librados: init done 2023-10-30T20:24:07.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fb993ade640 10 client.4294.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:07.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fb993ade640 15 client.4294.objecter _session_command_op_assign -1 1 2023-10-30T20:24:07.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fb993ade640 20 client.4294.objecter _get_session osd=-1 returning homeless 2023-10-30T20:24:07.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fb993ade640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fb993ade640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fb993ade640 1 -- 172.21.15.69:0/1022410054 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb98c13abd0 con 0x7fb98c13eb80 2023-10-30T20:24:07.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae7b7fe640 1 -- 172.21.15.69:0/1377797277 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae84010870 con 0x7fae940a8e40 2023-10-30T20:24:07.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae7b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:07.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae7b7fe640 10 monclient: dump: 2023-10-30T20:24:07.338 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.338 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.338 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.338 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.338 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.339 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.339 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.339 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.339 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.339 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae7b7fe640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae7b7fe640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae7b7fe640 1 -- 172.21.15.69:0/1377797277 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae840109e0 con 0x7fae940a8e40 2023-10-30T20:24:07.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae7b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae7b7fe640 1 -- 172.21.15.69:0/1377797277 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae84010da0 con 0x7fae940a8e40 2023-10-30T20:24:07.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae7b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae7b7fe640 10 monclient: dump: 2023-10-30T20:24:07.341 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.341 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.342 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.342 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.342 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.342 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.342 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.343 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.343 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.343 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae99621640 5 monclient: authenticate success, global_id 4300 2023-10-30T20:24:07.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae99621640 10 monclient: _renew_subs 2023-10-30T20:24:07.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae99621640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae99621640 1 -- 172.21.15.69:0/1377797277 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae941c8dc0 con 0x7fae940a8e40 2023-10-30T20:24:07.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae99621640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae99621640 10 monclient: _renew_subs 2023-10-30T20:24:07.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae99621640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae99621640 1 -- 172.21.15.69:0/1377797277 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae941d6890 con 0x7fae940a8e40 2023-10-30T20:24:07.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae917fa640 4 set_mon_vals no callback set 2023-10-30T20:24:07.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7fae99621640 1 librados: init done 2023-10-30T20:24:07.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.049+0000 7f4628d92640 1 --2- 172.21.15.69:0/3841363599 >> v2:172.21.15.69:6805/2300065874 conn(0x7f46100725d0 0x7f46100749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.050+0000 7f4628d92640 10 monclient: get_auth_request con 0x7f46100725d0 auth_method 0 2023-10-30T20:24:07.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.050+0000 7fae99621640 10 client.4300.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:07.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.050+0000 7fae99621640 15 client.4300.objecter _session_command_op_assign -1 1 2023-10-30T20:24:07.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.050+0000 7fae99621640 20 client.4300.objecter _get_session osd=-1 returning homeless 2023-10-30T20:24:07.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.050+0000 7fae99621640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.050+0000 7fae99621640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.050+0000 7fae99621640 1 -- 172.21.15.69:0/1377797277 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fae940a97b0 con 0x7fae940a8e40 2023-10-30T20:24:07.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.053+0000 7f4628d92640 1 --2- 172.21.15.69:0/3841363599 >> v2:172.21.15.69:6805/2300065874 conn(0x7f46100725d0 0x7f46100749f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.053+0000 7f4620ff9640 10 client.4274.objecter ms_handle_connect 0x7f46100725d0 2023-10-30T20:24:07.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.056+0000 7f0f6dffb640 1 -- 172.21.15.69:0/2087719441 <== osd.6 v2:172.21.15.183:6804/1046221745 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f0f6800c940 con 0x7f0f5c0725d0 2023-10-30T20:24:07.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.057+0000 7f0f6dffb640 10 client.4279.objecter ms_dispatch 0x7f0f78000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:24:07.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.057+0000 7f0f6dffb640 10 client.4279.objecter _finish_command 1 = system:0 2023-10-30T20:24:07.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.057+0000 7f0f6dffb640 20 client.4279.objecter put_session s=0x7f0f6800ccf0 osd=6 4 2023-10-30T20:24:07.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.057+0000 7f0f6dffb640 15 client.4279.objecter _session_command_op_remove 6 1 2023-10-30T20:24:07.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.068+0000 7f5912ffd640 1 -- 172.21.15.69:0/3480831939 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f5918003b30 con 0x7f591c0a8e40 2023-10-30T20:24:07.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.068+0000 7f5912ffd640 1 --2- 172.21.15.69:0/3480831939 >> v2:172.21.15.69:6800/1876307353 conn(0x7f590006c840 0x7f590006ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.069+0000 7f1cdbfff640 1 -- 172.21.15.69:0/3287870425 <== osd.3 v2:172.21.15.69:6809/1824616966 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f1ce40052d0 con 0x7f1ce0072620 2023-10-30T20:24:07.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.069+0000 7f1cdbfff640 10 client.4284.objecter ms_dispatch 0x7f1cf4000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:24:07.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.069+0000 7f1cdbfff640 10 client.4284.objecter _finish_command 1 = system:0 2023-10-30T20:24:07.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.069+0000 7f1cdbfff640 20 client.4284.objecter put_session s=0x7f1ce4012630 osd=3 4 2023-10-30T20:24:07.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.069+0000 7f1cdbfff640 15 client.4284.objecter _session_command_op_remove 3 1 2023-10-30T20:24:07.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.069+0000 7f59218b8640 1 --2- 172.21.15.69:0/3480831939 >> v2:172.21.15.69:6800/1876307353 conn(0x7f590006c840 0x7f590006ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.069+0000 7f59218b8640 10 monclient: get_auth_request con 0x7f590006c840 auth_method 0 2023-10-30T20:24:07.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.070+0000 7f59218b8640 1 --2- 172.21.15.69:0/3480831939 >> v2:172.21.15.69:6800/1876307353 conn(0x7f590006c840 0x7f590006ed00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f591c1444f0 tx=0x7f590c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.070+0000 7f5912ffd640 10 client.4293.objecter ms_handle_connect 0x7f590006c840 2023-10-30T20:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.071+0000 7f9beeffd640 1 -- 172.21.15.69:0/1950889256 <== osd.7 v2:172.21.15.183:6808/1361328034 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f9bf4005ce0 con 0x7f9bdc072620 2023-10-30T20:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.071+0000 7f9beeffd640 10 client.4271.objecter ms_dispatch 0x7f9bf8000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.071+0000 7f9beeffd640 10 client.4271.objecter _finish_command 1 = system:0 2023-10-30T20:24:07.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.071+0000 7f9beeffd640 20 client.4271.objecter put_session s=0x7f9bf4007270 osd=7 4 2023-10-30T20:24:07.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.071+0000 7f9beeffd640 15 client.4271.objecter _session_command_op_remove 7 1 2023-10-30T20:24:07.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 1 -- 172.21.15.69:0/3480831939 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f5918058130 con 0x7f591c0a8e40 2023-10-30T20:24:07.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 10 client.4293.objecter ms_dispatch 0x7f591c000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:07.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 3 client.4293.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:07.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 3 client.4293.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:07.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 10 client.4293.objecter checking command 1 2023-10-30T20:24:07.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 1 --2- 172.21.15.69:0/3480831939 >> v2:172.21.15.69:6813/431025540 conn(0x7f5900072650 0x7f5900074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 20 client.4293.objecter _get_session s=0x7f59180039c0 osd=1 3 2023-10-30T20:24:07.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 20 client.4293.objecter put_session s=0x7f59180039c0 osd=1 3 2023-10-30T20:24:07.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 15 client.4293.objecter _session_command_op_remove -1 1 2023-10-30T20:24:07.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 20 client.4293.objecter _get_session s=0x7f59180039c0 osd=1 3 2023-10-30T20:24:07.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 20 client.4293.objecter get_session s=0x7f59180039c0 osd=1 3 2023-10-30T20:24:07.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 15 client.4293.objecter _session_command_op_assign 1 1 2023-10-30T20:24:07.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 20 client.4293.objecter put_session s=0x7f59180039c0 osd=1 4 2023-10-30T20:24:07.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 10 client.4293.objecter _send_command 1 2023-10-30T20:24:07.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 1 -- 172.21.15.69:0/3480831939 --> v2:172.21.15.69:6813/431025540 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59180035f0 con 0x7f5900072650 2023-10-30T20:24:07.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 20 client.4293.objecter dump_active .. 0 homeless 2023-10-30T20:24:07.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 1 -- 172.21.15.69:0/3480831939 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5918053d70 con 0x7f591c0a8e40 2023-10-30T20:24:07.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.074+0000 7f5912ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T20:24:07.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.075+0000 7f59228ba640 1 --2- 172.21.15.69:0/3480831939 >> v2:172.21.15.69:6813/431025540 conn(0x7f5900072650 0x7f5900074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.075+0000 7f59228ba640 10 monclient: get_auth_request con 0x7f5900072650 auth_method 0 2023-10-30T20:24:07.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.076+0000 7fb9827fc640 1 -- 172.21.15.69:0/1022410054 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fb97c009990 con 0x7fb98c13eb80 2023-10-30T20:24:07.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.076+0000 7fb9827fc640 1 --2- 172.21.15.69:0/1022410054 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb97006c880 0x7fb97006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.077+0000 7fb99187a640 1 --2- 172.21.15.69:0/1022410054 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb97006c880 0x7fb97006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.077+0000 7fb99187a640 10 monclient: get_auth_request con 0x7fb97006c880 auth_method 0 2023-10-30T20:24:07.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.077+0000 7fb99187a640 1 --2- 172.21.15.69:0/1022410054 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb97006c880 0x7fb97006ed40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb988035000 tx=0x7fb98802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.077+0000 7fb9827fc640 10 client.4294.objecter ms_handle_connect 0x7fb97006c880 2023-10-30T20:24:07.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 1 -- 172.21.15.69:0/1022410054 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fb97c058e90 con 0x7fb98c13eb80 2023-10-30T20:24:07.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 10 client.4294.objecter ms_dispatch 0x7fb98c000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:07.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 3 client.4294.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:07.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 3 client.4294.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:07.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 10 client.4294.objecter checking command 1 2023-10-30T20:24:07.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 1 --2- 172.21.15.69:0/1022410054 >> v2:172.21.15.183:6800/4151811482 conn(0x7fb970072800 0x7fb970074c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 20 client.4294.objecter _get_session s=0x7fb970072150 osd=4 3 2023-10-30T20:24:07.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 20 client.4294.objecter put_session s=0x7fb970072150 osd=4 3 2023-10-30T20:24:07.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 15 client.4294.objecter _session_command_op_remove -1 1 2023-10-30T20:24:07.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 20 client.4294.objecter _get_session s=0x7fb970072150 osd=4 3 2023-10-30T20:24:07.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 20 client.4294.objecter get_session s=0x7fb970072150 osd=4 3 2023-10-30T20:24:07.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 15 client.4294.objecter _session_command_op_assign 4 1 2023-10-30T20:24:07.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 20 client.4294.objecter put_session s=0x7fb970072150 osd=4 4 2023-10-30T20:24:07.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 10 client.4294.objecter _send_command 1 2023-10-30T20:24:07.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 1 -- 172.21.15.69:0/1022410054 --> v2:172.21.15.183:6800/4151811482 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb970075290 con 0x7fb970072800 2023-10-30T20:24:07.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb9827fc640 20 client.4294.objecter dump_active .. 0 homeless 2023-10-30T20:24:07.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.078+0000 7fb99207b640 1 --2- 172.21.15.69:0/1022410054 >> v2:172.21.15.183:6800/4151811482 conn(0x7fb970072800 0x7fb970074c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.079+0000 7fb99207b640 10 monclient: get_auth_request con 0x7fb970072800 auth_method 0 2023-10-30T20:24:07.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.079+0000 7f59228ba640 1 --2- 172.21.15.69:0/3480831939 >> v2:172.21.15.69:6813/431025540 conn(0x7f5900072650 0x7f5900074a70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.079+0000 7f5912ffd640 10 client.4293.objecter ms_handle_connect 0x7f5900072650 2023-10-30T20:24:07.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.080+0000 7f0f7fa2d640 10 client.4279.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T20:24:07.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.080+0000 7f0f7fa2d640 15 client.4279.objecter _session_command_op_assign -1 2 2023-10-30T20:24:07.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.080+0000 7f0f7fa2d640 20 client.4279.objecter _get_session s=0x7f0f6800ccf0 osd=6 3 2023-10-30T20:24:07.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.080+0000 7f0f7fa2d640 20 client.4279.objecter put_session s=0x7f0f6800ccf0 osd=6 3 2023-10-30T20:24:07.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.080+0000 7f0f7fa2d640 20 client.4279.objecter _get_session s=0x7f0f6800ccf0 osd=6 3 2023-10-30T20:24:07.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.080+0000 7f0f7fa2d640 15 client.4279.objecter _session_command_op_remove -1 2 2023-10-30T20:24:07.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.080+0000 7f0f7fa2d640 20 client.4279.objecter get_session s=0x7f0f6800ccf0 osd=6 3 2023-10-30T20:24:07.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.080+0000 7f0f7fa2d640 15 client.4279.objecter _session_command_op_assign 6 2 2023-10-30T20:24:07.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.080+0000 7f0f7fa2d640 20 client.4279.objecter put_session s=0x7f0f6800ccf0 osd=6 4 2023-10-30T20:24:07.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.080+0000 7f0f7fa2d640 10 client.4279.objecter _send_command 2 2023-10-30T20:24:07.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.080+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2087719441 --> v2:172.21.15.183:6804/1046221745 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0f700cb110 con 0x7f0f5c0725d0 2023-10-30T20:24:07.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.082+0000 7fb99207b640 1 -- 172.21.15.69:0/1022410054 >> v2:172.21.15.183:6800/4151811482 conn(0x7fb970072800 msgr2=0x7fb970074c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T20:24:07.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.082+0000 7fb99207b640 1 -- 172.21.15.69:0/1022410054 >> v2:172.21.15.183:6800/4151811482 conn(0x7fb970072800 msgr2=0x7fb970074c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T20:24:07.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.082+0000 7fb99207b640 1 --2- 172.21.15.69:0/1022410054 >> v2:172.21.15.183:6800/4151811482 conn(0x7fb970072800 0x7fb970074c20 crc :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T20:24:07.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.082+0000 7fb99207b640 1 --2- 172.21.15.69:0/1022410054 >> v2:172.21.15.183:6800/4151811482 conn(0x7fb970072800 0x7fb970074c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T20:24:07.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.093+0000 7f0f6dffb640 1 -- 172.21.15.69:0/2087719441 <== osd.6 v2:172.21.15.183:6804/1046221745 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0f78069190 con 0x7f0f5c0725d0 2023-10-30T20:24:07.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.093+0000 7f0f6dffb640 10 client.4279.objecter ms_dispatch 0x7f0f78000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:24:07.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.093+0000 7f0f6dffb640 10 client.4279.objecter _finish_command 2 = system:0 2023-10-30T20:24:07.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.093+0000 7f0f6dffb640 20 client.4279.objecter put_session s=0x7f0f6800ccf0 osd=6 4 2023-10-30T20:24:07.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.093+0000 7f0f6dffb640 15 client.4279.objecter _session_command_op_remove 6 2 2023-10-30T20:24:07.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 10 librados: watch_flush enter 2023-10-30T20:24:07.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 10 librados: watch_flush exit 2023-10-30T20:24:07.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 10 client.4279.objecter close_session for osd.6 2023-10-30T20:24:07.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2087719441 >> v2:172.21.15.183:6804/1046221745 conn(0x7f0f5c0725d0 msgr2=0x7f0f5c0749f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 1 --2- 172.21.15.69:0/2087719441 >> v2:172.21.15.183:6804/1046221745 conn(0x7f0f5c0725d0 0x7f0f5c0749f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 20 client.4279.objecter put_session s=0x7f0f6800ccf0 osd=6 1 2023-10-30T20:24:07.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 20 client.4279.objecter shutdown clearing up homeless session... 2023-10-30T20:24:07.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 10 client.4279.objecter successfully canceled tick 2023-10-30T20:24:07.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2087719441 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0f5c06c7c0 msgr2=0x7f0f5c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 1 --2- 172.21.15.69:0/2087719441 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0f5c06c7c0 0x7f0f5c06ec80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0f74036000 tx=0x7f0f7403b040 comp rx=0 tx=0).stop 2023-10-30T20:24:07.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 10 monclient: shutdown 2023-10-30T20:24:07.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2087719441 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f70033600 msgr2=0x7f0f700c7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 1 --2- 172.21.15.69:0/2087719441 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f70033600 0x7f0f700c7480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0f6800e9b0 tx=0x7f0f6800edb0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7dfca640 1 -- 172.21.15.69:0/2087719441 reap_dead start 2023-10-30T20:24:07.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2087719441 shutdown_connections 2023-10-30T20:24:07.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2087719441 >> 172.21.15.69:0/2087719441 conn(0x7f0f70006030 msgr2=0x7f0f70004de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2087719441 shutdown_connections 2023-10-30T20:24:07.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 1 -- 172.21.15.69:0/2087719441 wait complete. 2023-10-30T20:24:07.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.094+0000 7f0f7fa2d640 1 librados: shutdown 2023-10-30T20:24:07.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.096+0000 7feeaffff640 1 -- 172.21.15.69:0/553467047 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7feea400f3c0 con 0x7feeb80ab6b0 2023-10-30T20:24:07.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.097+0000 7feeaffff640 1 --2- 172.21.15.69:0/553467047 >> v2:172.21.15.69:6800/1876307353 conn(0x7feea006c810 0x7feea006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.097+0000 7feeb77fe640 1 --2- 172.21.15.69:0/553467047 >> v2:172.21.15.69:6800/1876307353 conn(0x7feea006c810 0x7feea006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.097+0000 7feeb77fe640 10 monclient: get_auth_request con 0x7feea006c810 auth_method 0 2023-10-30T20:24:07.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.098+0000 7feeb77fe640 1 --2- 172.21.15.69:0/553467047 >> v2:172.21.15.69:6800/1876307353 conn(0x7feea006c810 0x7feea006ecd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feea8035000 tx=0x7feea8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.098+0000 7feeaffff640 10 client.4303.objecter ms_handle_connect 0x7feea006c810 2023-10-30T20:24:07.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.101+0000 7f1cf984b640 10 client.4284.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T20:24:07.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.101+0000 7f1cf984b640 15 client.4284.objecter _session_command_op_assign -1 2 2023-10-30T20:24:07.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.101+0000 7f1cf984b640 20 client.4284.objecter _get_session s=0x7f1ce4012630 osd=3 3 2023-10-30T20:24:07.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.101+0000 7f1cf984b640 20 client.4284.objecter put_session s=0x7f1ce4012630 osd=3 3 2023-10-30T20:24:07.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.101+0000 7f1cf984b640 20 client.4284.objecter _get_session s=0x7f1ce4012630 osd=3 3 2023-10-30T20:24:07.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.101+0000 7f1cf984b640 15 client.4284.objecter _session_command_op_remove -1 2 2023-10-30T20:24:07.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.101+0000 7f1cf984b640 20 client.4284.objecter get_session s=0x7f1ce4012630 osd=3 3 2023-10-30T20:24:07.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.101+0000 7f1cf984b640 15 client.4284.objecter _session_command_op_assign 3 2 2023-10-30T20:24:07.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.101+0000 7f1cf984b640 20 client.4284.objecter put_session s=0x7f1ce4012630 osd=3 4 2023-10-30T20:24:07.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.101+0000 7f1cf984b640 10 client.4284.objecter _send_command 2 2023-10-30T20:24:07.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.101+0000 7f1cf984b640 1 -- 172.21.15.69:0/3287870425 --> v2:172.21.15.69:6809/1824616966 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1cec0074c0 con 0x7f1ce0072620 2023-10-30T20:24:07.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f9c00541640 10 client.4271.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T20:24:07.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f9c00541640 15 client.4271.objecter _session_command_op_assign -1 2 2023-10-30T20:24:07.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f9c00541640 20 client.4271.objecter _get_session s=0x7f9bf4007270 osd=7 3 2023-10-30T20:24:07.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f9c00541640 20 client.4271.objecter put_session s=0x7f9bf4007270 osd=7 3 2023-10-30T20:24:07.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f9c00541640 20 client.4271.objecter _get_session s=0x7f9bf4007270 osd=7 3 2023-10-30T20:24:07.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f9c00541640 15 client.4271.objecter _session_command_op_remove -1 2 2023-10-30T20:24:07.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f9c00541640 20 client.4271.objecter get_session s=0x7f9bf4007270 osd=7 3 2023-10-30T20:24:07.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f9c00541640 15 client.4271.objecter _session_command_op_assign 7 2 2023-10-30T20:24:07.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f9c00541640 20 client.4271.objecter put_session s=0x7f9bf4007270 osd=7 4 2023-10-30T20:24:07.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f9c00541640 10 client.4271.objecter _send_command 2 2023-10-30T20:24:07.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f9c00541640 1 -- 172.21.15.69:0/1950889256 --> v2:172.21.15.183:6808/1361328034 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9be00011f0 con 0x7f9bdc072620 2023-10-30T20:24:07.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f4620ff9640 1 -- 172.21.15.69:0/3841363599 <== osd.2 v2:172.21.15.69:6805/2300065874 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f4618032860 con 0x7f46100725d0 2023-10-30T20:24:07.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f4620ff9640 10 client.4274.objecter ms_dispatch 0x7f4624000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:24:07.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f4620ff9640 10 client.4274.objecter _finish_command 1 = system:0 2023-10-30T20:24:07.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f4620ff9640 20 client.4274.objecter put_session s=0x7f4618032c30 osd=2 4 2023-10-30T20:24:07.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.102+0000 7f4620ff9640 15 client.4274.objecter _session_command_op_remove 2 1 2023-10-30T20:24:07.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.105+0000 7feeaffff640 1 -- 172.21.15.69:0/553467047 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7feea4058b40 con 0x7feeb80ab6b0 2023-10-30T20:24:07.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.105+0000 7feeaffff640 10 client.4303.objecter ms_dispatch 0x7feeb8000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:07.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.105+0000 7feeaffff640 3 client.4303.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:07.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.105+0000 7feeaffff640 3 client.4303.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:07.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.105+0000 7feeaffff640 10 client.4303.objecter checking command 1 2023-10-30T20:24:07.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.105+0000 7fb9827fc640 1 -- 172.21.15.69:0/1022410054 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb97c054ad0 con 0x7fb98c13eb80 2023-10-30T20:24:07.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.105+0000 7fb9827fc640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T20:24:07.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.105+0000 7feeaffff640 1 --2- 172.21.15.69:0/553467047 >> v2:172.21.15.69:6801/2068724062 conn(0x7feea0072620 0x7feea0074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.105+0000 7feeaffff640 20 client.4303.objecter _get_session s=0x7feea4010da0 osd=0 3 2023-10-30T20:24:07.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.105+0000 7feeaffff640 20 client.4303.objecter put_session s=0x7feea4010da0 osd=0 3 2023-10-30T20:24:07.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.105+0000 7feeaffff640 15 client.4303.objecter _session_command_op_remove -1 1 2023-10-30T20:24:07.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.105+0000 7feeaffff640 20 client.4303.objecter _get_session s=0x7feea4010da0 osd=0 3 2023-10-30T20:24:07.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.106+0000 7feeaffff640 20 client.4303.objecter get_session s=0x7feea4010da0 osd=0 3 2023-10-30T20:24:07.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.106+0000 7feeaffff640 15 client.4303.objecter _session_command_op_assign 0 1 2023-10-30T20:24:07.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.106+0000 7feeaffff640 20 client.4303.objecter put_session s=0x7feea4010da0 osd=0 4 2023-10-30T20:24:07.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.106+0000 7feeaffff640 10 client.4303.objecter _send_command 1 2023-10-30T20:24:07.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.106+0000 7feeb7fff640 1 --2- 172.21.15.69:0/553467047 >> v2:172.21.15.69:6801/2068724062 conn(0x7feea0072620 0x7feea0074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.106+0000 7feeaffff640 1 -- 172.21.15.69:0/553467047 --> v2:172.21.15.69:6801/2068724062 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feea4010870 con 0x7feea0072620 2023-10-30T20:24:07.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.106+0000 7feeaffff640 20 client.4303.objecter dump_active .. 0 homeless 2023-10-30T20:24:07.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.106+0000 7feeb7fff640 10 monclient: get_auth_request con 0x7feea0072620 auth_method 0 2023-10-30T20:24:07.387 INFO:teuthology.orchestra.run.smithi069.stdout:64424509446 2023-10-30T20:24:07.388 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T20:24:07.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cdbfff640 1 -- 172.21.15.69:0/3287870425 <== osd.3 v2:172.21.15.69:6809/1824616966 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1ce8001280 con 0x7f1ce0072620 2023-10-30T20:24:07.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cdbfff640 10 client.4284.objecter ms_dispatch 0x7f1cf4000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:24:07.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cdbfff640 10 client.4284.objecter _finish_command 2 = system:0 2023-10-30T20:24:07.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cdbfff640 20 client.4284.objecter put_session s=0x7f1ce4012630 osd=3 4 2023-10-30T20:24:07.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cdbfff640 15 client.4284.objecter _session_command_op_remove 3 2 2023-10-30T20:24:07.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cf984b640 10 librados: watch_flush enter 2023-10-30T20:24:07.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cf984b640 10 librados: watch_flush exit 2023-10-30T20:24:07.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cf984b640 10 client.4284.objecter close_session for osd.3 2023-10-30T20:24:07.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cf984b640 1 -- 172.21.15.69:0/3287870425 >> v2:172.21.15.69:6809/1824616966 conn(0x7f1ce0072620 msgr2=0x7f1ce0074a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cf984b640 1 --2- 172.21.15.69:0/3287870425 >> v2:172.21.15.69:6809/1824616966 conn(0x7f1ce0072620 0x7f1ce0074a40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cf984b640 20 client.4284.objecter put_session s=0x7f1ce4012630 osd=3 1 2023-10-30T20:24:07.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cf984b640 20 client.4284.objecter shutdown clearing up homeless session... 2023-10-30T20:24:07.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cf984b640 10 client.4284.objecter successfully canceled tick 2023-10-30T20:24:07.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cf984b640 1 -- 172.21.15.69:0/3287870425 >> v2:172.21.15.69:6800/1876307353 conn(0x7f1ce006c810 msgr2=0x7f1ce006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.109+0000 7f1cf984b640 1 --2- 172.21.15.69:0/3287870425 >> v2:172.21.15.69:6800/1876307353 conn(0x7f1ce006c810 0x7f1ce006ecd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1cec031280 tx=0x7f1cf40aa000 comp rx=0 tx=0).stop 2023-10-30T20:24:07.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.110+0000 7f1cf984b640 10 monclient: shutdown 2023-10-30T20:24:07.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.110+0000 7f1cf984b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.110+0000 7f1cf984b640 1 -- 172.21.15.69:0/3287870425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cec0070e0 msgr2=0x7f1cec02d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.110+0000 7f1cf984b640 1 --2- 172.21.15.69:0/3287870425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cec0070e0 0x7f1cec02d7f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400c480 tx=0x7f1ce400c940 comp rx=0 tx=0).stop 2023-10-30T20:24:07.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.110+0000 7f1cf37fe640 1 -- 172.21.15.69:0/3287870425 reap_dead start 2023-10-30T20:24:07.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.110+0000 7f1cf984b640 1 -- 172.21.15.69:0/3287870425 shutdown_connections 2023-10-30T20:24:07.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.110+0000 7f1cf984b640 1 -- 172.21.15.69:0/3287870425 >> 172.21.15.69:0/3287870425 conn(0x7f1cec006030 msgr2=0x7f1cec033600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.110+0000 7f1cf984b640 1 -- 172.21.15.69:0/3287870425 shutdown_connections 2023-10-30T20:24:07.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.110+0000 7f1cf984b640 1 -- 172.21.15.69:0/3287870425 wait complete. 2023-10-30T20:24:07.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.110+0000 7f1cf984b640 1 librados: shutdown 2023-10-30T20:24:07.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.110+0000 7feeaffff640 10 client.4303.objecter ms_handle_connect 0x7feea0072620 2023-10-30T20:24:07.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.110+0000 7feeb7fff640 1 --2- 172.21.15.69:0/553467047 >> v2:172.21.15.69:6801/2068724062 conn(0x7feea0072620 0x7feea0074a40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.395 INFO:teuthology.orchestra.run.smithi069.stdout:68719476742 2023-10-30T20:24:07.395 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T20:24:07.395 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.122+0000 7fae7b7fe640 1 -- 172.21.15.69:0/1377797277 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fae8400f3c0 con 0x7fae940a8e40 2023-10-30T20:24:07.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.122+0000 7feeaffff640 1 -- 172.21.15.69:0/553467047 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7feeb80abc60 con 0x7feeb80ab6b0 2023-10-30T20:24:07.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.122+0000 7feeaffff640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T20:24:07.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.122+0000 7fae7b7fe640 1 --2- 172.21.15.69:0/1377797277 >> v2:172.21.15.69:6800/1876307353 conn(0x7fae8006c810 0x7fae8006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.122+0000 7fae927fc640 1 --2- 172.21.15.69:0/1377797277 >> v2:172.21.15.69:6800/1876307353 conn(0x7fae8006c810 0x7fae8006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.123+0000 7fae927fc640 10 monclient: get_auth_request con 0x7fae8006c810 auth_method 0 2023-10-30T20:24:07.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.123+0000 7fae7b7fe640 10 client.4300.objecter ms_handle_connect 0x7fae8006c810 2023-10-30T20:24:07.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.123+0000 7fae927fc640 1 --2- 172.21.15.69:0/1377797277 >> v2:172.21.15.69:6800/1876307353 conn(0x7fae8006c810 0x7fae8006ecd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fae88036000 tx=0x7fae8803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.124+0000 7f5912ffd640 1 -- 172.21.15.69:0/3480831939 <== osd.1 v2:172.21.15.69:6813/431025540 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f59180035f0 con 0x7f5900072650 2023-10-30T20:24:07.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.124+0000 7f5912ffd640 10 client.4293.objecter ms_dispatch 0x7f591c000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:24:07.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.124+0000 7f5912ffd640 10 client.4293.objecter _finish_command 1 = system:0 2023-10-30T20:24:07.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.124+0000 7f5912ffd640 20 client.4293.objecter put_session s=0x7f59180039c0 osd=1 4 2023-10-30T20:24:07.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.124+0000 7f5912ffd640 15 client.4293.objecter _session_command_op_remove 1 1 2023-10-30T20:24:07.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 1 -- 172.21.15.69:0/1377797277 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fae840583f0 con 0x7fae940a8e40 2023-10-30T20:24:07.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 10 client.4300.objecter ms_dispatch 0x7fae94000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:07.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 3 client.4300.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:07.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 3 client.4300.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:07.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 10 client.4300.objecter checking command 1 2023-10-30T20:24:07.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 1 --2- 172.21.15.69:0/1377797277 >> v2:172.21.15.183:6812/2276632263 conn(0x7fae80072620 0x7fae80074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.401 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 20 client.4300.objecter _get_session s=0x7fae84010da0 osd=5 3 2023-10-30T20:24:07.401 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 20 client.4300.objecter put_session s=0x7fae84010da0 osd=5 3 2023-10-30T20:24:07.401 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 15 client.4300.objecter _session_command_op_remove -1 1 2023-10-30T20:24:07.401 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 20 client.4300.objecter _get_session s=0x7fae84010da0 osd=5 3 2023-10-30T20:24:07.401 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 20 client.4300.objecter get_session s=0x7fae84010da0 osd=5 3 2023-10-30T20:24:07.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 15 client.4300.objecter _session_command_op_assign 5 1 2023-10-30T20:24:07.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 20 client.4300.objecter put_session s=0x7fae84010da0 osd=5 4 2023-10-30T20:24:07.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 10 client.4300.objecter _send_command 1 2023-10-30T20:24:07.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 1 -- 172.21.15.69:0/1377797277 --> v2:172.21.15.183:6812/2276632263 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae84010870 con 0x7fae80072620 2023-10-30T20:24:07.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 20 client.4300.objecter dump_active .. 0 homeless 2023-10-30T20:24:07.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 1 -- 172.21.15.69:0/1377797277 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae84054730 con 0x7fae940a8e40 2023-10-30T20:24:07.403 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.125+0000 7fae7b7fe640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T20:24:07.403 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f462a7f5640 10 client.4274.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T20:24:07.403 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f462a7f5640 15 client.4274.objecter _session_command_op_assign -1 2 2023-10-30T20:24:07.403 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f462a7f5640 20 client.4274.objecter _get_session s=0x7f4618032c30 osd=2 3 2023-10-30T20:24:07.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f462a7f5640 20 client.4274.objecter put_session s=0x7f4618032c30 osd=2 3 2023-10-30T20:24:07.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f462a7f5640 20 client.4274.objecter _get_session s=0x7f4618032c30 osd=2 3 2023-10-30T20:24:07.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f462a7f5640 15 client.4274.objecter _session_command_op_remove -1 2 2023-10-30T20:24:07.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f462a7f5640 20 client.4274.objecter get_session s=0x7f4618032c30 osd=2 3 2023-10-30T20:24:07.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f462a7f5640 15 client.4274.objecter _session_command_op_assign 2 2 2023-10-30T20:24:07.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f462a7f5640 20 client.4274.objecter put_session s=0x7f4618032c30 osd=2 4 2023-10-30T20:24:07.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f462a7f5640 10 client.4274.objecter _send_command 2 2023-10-30T20:24:07.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f462a7f5640 1 -- 172.21.15.69:0/3841363599 --> v2:172.21.15.69:6805/2300065874 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f46040011c0 con 0x7f46100725d0 2023-10-30T20:24:07.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7fae937fe640 1 --2- 172.21.15.69:0/1377797277 >> v2:172.21.15.183:6812/2276632263 conn(0x7fae80072620 0x7fae80074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7fae937fe640 10 monclient: get_auth_request con 0x7fae80072620 auth_method 0 2023-10-30T20:24:07.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f9beeffd640 1 -- 172.21.15.69:0/1950889256 <== osd.7 v2:172.21.15.183:6808/1361328034 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9be00011f0 con 0x7f9bdc072620 2023-10-30T20:24:07.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f9beeffd640 10 client.4271.objecter ms_dispatch 0x7f9bf8000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:24:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f9beeffd640 10 client.4271.objecter _finish_command 2 = system:0 2023-10-30T20:24:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f9beeffd640 20 client.4271.objecter put_session s=0x7f9bf4007270 osd=7 4 2023-10-30T20:24:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.126+0000 7f9beeffd640 15 client.4271.objecter _session_command_op_remove 7 2 2023-10-30T20:24:07.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 10 librados: watch_flush enter 2023-10-30T20:24:07.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 10 librados: watch_flush exit 2023-10-30T20:24:07.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 10 client.4271.objecter close_session for osd.7 2023-10-30T20:24:07.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 1 -- 172.21.15.69:0/1950889256 >> v2:172.21.15.183:6808/1361328034 conn(0x7f9bdc072620 msgr2=0x7f9bdc074a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 1 --2- 172.21.15.69:0/1950889256 >> v2:172.21.15.183:6808/1361328034 conn(0x7f9bdc072620 0x7f9bdc074a40 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 20 client.4271.objecter put_session s=0x7f9bf4007270 osd=7 1 2023-10-30T20:24:07.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 20 client.4271.objecter shutdown clearing up homeless session... 2023-10-30T20:24:07.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 10 client.4271.objecter successfully canceled tick 2023-10-30T20:24:07.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 1 -- 172.21.15.69:0/1950889256 >> v2:172.21.15.69:6800/1876307353 conn(0x7f9bdc06c810 msgr2=0x7f9bdc06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 1 --2- 172.21.15.69:0/1950889256 >> v2:172.21.15.69:6800/1876307353 conn(0x7f9bdc06c810 0x7f9bdc06ecd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9be8036000 tx=0x7f9be8030000 comp rx=0 tx=0).stop 2023-10-30T20:24:07.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 10 monclient: shutdown 2023-10-30T20:24:07.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 1 -- 172.21.15.69:0/1950889256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bf0030b10 msgr2=0x7f9bf002d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 1 --2- 172.21.15.69:0/1950889256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bf0030b10 0x7f9bf002d580 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400c560 tx=0x7f9bf400ca20 comp rx=0 tx=0).stop 2023-10-30T20:24:07.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9bfeade640 1 -- 172.21.15.69:0/1950889256 reap_dead start 2023-10-30T20:24:07.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 1 -- 172.21.15.69:0/1950889256 shutdown_connections 2023-10-30T20:24:07.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 1 -- 172.21.15.69:0/1950889256 >> 172.21.15.69:0/1950889256 conn(0x7f9bf0006030 msgr2=0x7f9bf0004de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 1 -- 172.21.15.69:0/1950889256 shutdown_connections 2023-10-30T20:24:07.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 1 -- 172.21.15.69:0/1950889256 wait complete. 2023-10-30T20:24:07.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.127+0000 7f9c00541640 1 librados: shutdown 2023-10-30T20:24:07.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.130+0000 7fae7b7fe640 10 client.4300.objecter ms_handle_connect 0x7fae80072620 2023-10-30T20:24:07.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.130+0000 7fae937fe640 1 --2- 172.21.15.69:0/1377797277 >> v2:172.21.15.183:6812/2276632263 conn(0x7fae80072620 0x7fae80074a40 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f4620ff9640 1 -- 172.21.15.69:0/3841363599 <== osd.2 v2:172.21.15.69:6805/2300065874 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f46040011c0 con 0x7f46100725d0 2023-10-30T20:24:07.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f4620ff9640 10 client.4274.objecter ms_dispatch 0x7f4624000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:24:07.414 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f4620ff9640 10 client.4274.objecter _finish_command 2 = system:0 2023-10-30T20:24:07.414 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f4620ff9640 20 client.4274.objecter put_session s=0x7f4618032c30 osd=2 4 2023-10-30T20:24:07.414 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f4620ff9640 15 client.4274.objecter _session_command_op_remove 2 2 2023-10-30T20:24:07.414 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 10 librados: watch_flush enter 2023-10-30T20:24:07.414 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 10 librados: watch_flush exit 2023-10-30T20:24:07.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 10 client.4274.objecter close_session for osd.2 2023-10-30T20:24:07.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 1 -- 172.21.15.69:0/3841363599 >> v2:172.21.15.69:6805/2300065874 conn(0x7f46100725d0 msgr2=0x7f46100749f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 1 --2- 172.21.15.69:0/3841363599 >> v2:172.21.15.69:6805/2300065874 conn(0x7f46100725d0 0x7f46100749f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 20 client.4274.objecter put_session s=0x7f4618032c30 osd=2 1 2023-10-30T20:24:07.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 20 client.4274.objecter shutdown clearing up homeless session... 2023-10-30T20:24:07.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 10 client.4274.objecter successfully canceled tick 2023-10-30T20:24:07.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 1 -- 172.21.15.69:0/3841363599 >> v2:172.21.15.69:6800/1876307353 conn(0x7f461006c7c0 msgr2=0x7f461006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 1 --2- 172.21.15.69:0/3841363599 >> v2:172.21.15.69:6800/1876307353 conn(0x7f461006c7c0 0x7f461006ec80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f46241478a0 tx=0x7f4614008040 comp rx=0 tx=0).stop 2023-10-30T20:24:07.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 10 monclient: shutdown 2023-10-30T20:24:07.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 1 -- 172.21.15.69:0/3841363599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4624071710 msgr2=0x7f46240aa840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 1 --2- 172.21.15.69:0/3841363599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4624071710 0x7f46240aa840 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4618033000 tx=0x7f46180046c0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f4628d92640 1 -- 172.21.15.69:0/3841363599 reap_dead start 2023-10-30T20:24:07.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 1 -- 172.21.15.69:0/3841363599 shutdown_connections 2023-10-30T20:24:07.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.139+0000 7f462a7f5640 1 -- 172.21.15.69:0/3841363599 >> 172.21.15.69:0/3841363599 conn(0x7f4624069d50 msgr2=0x7f46240ae3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.140+0000 7f462a7f5640 1 -- 172.21.15.69:0/3841363599 shutdown_connections 2023-10-30T20:24:07.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.140+0000 7f462a7f5640 1 -- 172.21.15.69:0/3841363599 wait complete. 2023-10-30T20:24:07.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.140+0000 7f462a7f5640 1 librados: shutdown 2023-10-30T20:24:07.418 INFO:teuthology.orchestra.run.smithi069.stdout:60129542148 2023-10-30T20:24:07.419 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T20:24:07.419 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.147+0000 7f592431d640 10 client.4293.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T20:24:07.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.147+0000 7f592431d640 15 client.4293.objecter _session_command_op_assign -1 2 2023-10-30T20:24:07.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.147+0000 7f592431d640 20 client.4293.objecter _get_session s=0x7f59180039c0 osd=1 3 2023-10-30T20:24:07.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.147+0000 7f592431d640 20 client.4293.objecter put_session s=0x7f59180039c0 osd=1 3 2023-10-30T20:24:07.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.147+0000 7f592431d640 20 client.4293.objecter _get_session s=0x7f59180039c0 osd=1 3 2023-10-30T20:24:07.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.147+0000 7f592431d640 15 client.4293.objecter _session_command_op_remove -1 2 2023-10-30T20:24:07.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.147+0000 7f592431d640 20 client.4293.objecter get_session s=0x7f59180039c0 osd=1 3 2023-10-30T20:24:07.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.147+0000 7f592431d640 15 client.4293.objecter _session_command_op_assign 1 2 2023-10-30T20:24:07.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.147+0000 7f592431d640 20 client.4293.objecter put_session s=0x7f59180039c0 osd=1 4 2023-10-30T20:24:07.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.147+0000 7f592431d640 10 client.4293.objecter _send_command 2 2023-10-30T20:24:07.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.147+0000 7f592431d640 1 -- 172.21.15.69:0/3480831939 --> v2:172.21.15.69:6813/431025540 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f591c13f6d0 con 0x7f5900072650 2023-10-30T20:24:07.422 INFO:teuthology.orchestra.run.smithi069.stdout:60129542149 2023-10-30T20:24:07.423 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T20:24:07.423 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f5912ffd640 1 -- 172.21.15.69:0/3480831939 <== osd.1 v2:172.21.15.69:6813/431025540 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f591402cb40 con 0x7f5900072650 2023-10-30T20:24:07.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f5912ffd640 10 client.4293.objecter ms_dispatch 0x7f591c000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:24:07.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f5912ffd640 10 client.4293.objecter _finish_command 2 = system:0 2023-10-30T20:24:07.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f5912ffd640 20 client.4293.objecter put_session s=0x7f59180039c0 osd=1 4 2023-10-30T20:24:07.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f5912ffd640 15 client.4293.objecter _session_command_op_remove 1 2 2023-10-30T20:24:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 10 librados: watch_flush enter 2023-10-30T20:24:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 10 librados: watch_flush exit 2023-10-30T20:24:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 10 client.4293.objecter close_session for osd.1 2023-10-30T20:24:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 1 -- 172.21.15.69:0/3480831939 >> v2:172.21.15.69:6813/431025540 conn(0x7f5900072650 msgr2=0x7f5900074a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 1 --2- 172.21.15.69:0/3480831939 >> v2:172.21.15.69:6813/431025540 conn(0x7f5900072650 0x7f5900074a70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 20 client.4293.objecter put_session s=0x7f59180039c0 osd=1 1 2023-10-30T20:24:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 20 client.4293.objecter shutdown clearing up homeless session... 2023-10-30T20:24:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 10 client.4293.objecter successfully canceled tick 2023-10-30T20:24:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 1 -- 172.21.15.69:0/3480831939 >> v2:172.21.15.69:6800/1876307353 conn(0x7f590006c840 msgr2=0x7f590006ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 1 --2- 172.21.15.69:0/3480831939 >> v2:172.21.15.69:6800/1876307353 conn(0x7f590006c840 0x7f590006ed00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f591c1444f0 tx=0x7f590c009040 comp rx=0 tx=0).stop 2023-10-30T20:24:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 10 monclient: shutdown 2023-10-30T20:24:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 1 -- 172.21.15.69:0/3480831939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f591c0a8e40 msgr2=0x7f591c11f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 1 --2- 172.21.15.69:0/3480831939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f591c0a8e40 0x7f591c11f280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f591800c470 tx=0x7f591800c930 comp rx=0 tx=0).stop 2023-10-30T20:24:07.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f59228ba640 1 -- 172.21.15.69:0/3480831939 reap_dead start 2023-10-30T20:24:07.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 1 -- 172.21.15.69:0/3480831939 shutdown_connections 2023-10-30T20:24:07.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 1 -- 172.21.15.69:0/3480831939 >> 172.21.15.69:0/3480831939 conn(0x7f591c0b7180 msgr2=0x7f591c0b4250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 1 -- 172.21.15.69:0/3480831939 shutdown_connections 2023-10-30T20:24:07.430 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 1 -- 172.21.15.69:0/3480831939 wait complete. 2023-10-30T20:24:07.430 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.153+0000 7f592431d640 1 librados: shutdown 2023-10-30T20:24:07.430 INFO:teuthology.orchestra.run.smithi069.stdout:68719476741 2023-10-30T20:24:07.430 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T20:24:07.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.167+0000 7feeaffff640 1 -- 172.21.15.69:0/553467047 <== osd.0 v2:172.21.15.69:6801/2068724062 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7feea4010870 con 0x7feea0072620 2023-10-30T20:24:07.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.167+0000 7feeaffff640 10 client.4303.objecter ms_dispatch 0x7feeb8000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:24:07.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.167+0000 7feeaffff640 10 client.4303.objecter _finish_command 1 = system:0 2023-10-30T20:24:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.167+0000 7feeaffff640 20 client.4303.objecter put_session s=0x7feea4010da0 osd=0 4 2023-10-30T20:24:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.167+0000 7feeaffff640 15 client.4303.objecter _session_command_op_remove 0 1 2023-10-30T20:24:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.171+0000 7fae7b7fe640 1 -- 172.21.15.69:0/1377797277 <== osd.5 v2:172.21.15.183:6812/2276632263 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fae84010870 con 0x7fae80072620 2023-10-30T20:24:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.171+0000 7fae7b7fe640 10 client.4300.objecter ms_dispatch 0x7fae94000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:24:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.171+0000 7fae7b7fe640 10 client.4300.objecter _finish_command 1 = system:0 2023-10-30T20:24:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.171+0000 7fae7b7fe640 20 client.4300.objecter put_session s=0x7fae84010da0 osd=5 4 2023-10-30T20:24:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.171+0000 7fae7b7fe640 15 client.4300.objecter _session_command_op_remove 5 1 2023-10-30T20:24:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.181+0000 7feebe11d640 10 client.4303.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T20:24:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.181+0000 7feebe11d640 15 client.4303.objecter _session_command_op_assign -1 2 2023-10-30T20:24:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.181+0000 7feebe11d640 20 client.4303.objecter _get_session s=0x7feea4010da0 osd=0 3 2023-10-30T20:24:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.181+0000 7feebe11d640 20 client.4303.objecter put_session s=0x7feea4010da0 osd=0 3 2023-10-30T20:24:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.181+0000 7feebe11d640 20 client.4303.objecter _get_session s=0x7feea4010da0 osd=0 3 2023-10-30T20:24:07.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.181+0000 7feebe11d640 15 client.4303.objecter _session_command_op_remove -1 2 2023-10-30T20:24:07.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.181+0000 7feebe11d640 20 client.4303.objecter get_session s=0x7feea4010da0 osd=0 3 2023-10-30T20:24:07.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.181+0000 7feebe11d640 15 client.4303.objecter _session_command_op_assign 0 2 2023-10-30T20:24:07.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.181+0000 7feebe11d640 20 client.4303.objecter put_session s=0x7feea4010da0 osd=0 4 2023-10-30T20:24:07.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.181+0000 7feebe11d640 10 client.4303.objecter _send_command 2 2023-10-30T20:24:07.436 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.181+0000 7feebe11d640 1 -- 172.21.15.69:0/553467047 --> v2:172.21.15.69:6801/2068724062 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feeb8142930 con 0x7feea0072620 2023-10-30T20:24:07.436 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.185+0000 7fae99621640 10 client.4300.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T20:24:07.436 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.185+0000 7fae99621640 15 client.4300.objecter _session_command_op_assign -1 2 2023-10-30T20:24:07.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.185+0000 7fae99621640 20 client.4300.objecter _get_session s=0x7fae84010da0 osd=5 3 2023-10-30T20:24:07.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.185+0000 7fae99621640 20 client.4300.objecter put_session s=0x7fae84010da0 osd=5 3 2023-10-30T20:24:07.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.185+0000 7fae99621640 20 client.4300.objecter _get_session s=0x7fae84010da0 osd=5 3 2023-10-30T20:24:07.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.185+0000 7fae99621640 15 client.4300.objecter _session_command_op_remove -1 2 2023-10-30T20:24:07.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.185+0000 7fae99621640 20 client.4300.objecter get_session s=0x7fae84010da0 osd=5 3 2023-10-30T20:24:07.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.185+0000 7fae99621640 15 client.4300.objecter _session_command_op_assign 5 2 2023-10-30T20:24:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.185+0000 7fae99621640 20 client.4300.objecter put_session s=0x7fae84010da0 osd=5 4 2023-10-30T20:24:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.185+0000 7fae99621640 10 client.4300.objecter _send_command 2 2023-10-30T20:24:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.185+0000 7fae99621640 1 -- 172.21.15.69:0/1377797277 --> v2:172.21.15.183:6812/2276632263 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fae940a9440 con 0x7fae80072620 2023-10-30T20:24:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feeaffff640 1 -- 172.21.15.69:0/553467047 <== osd.0 v2:172.21.15.69:6801/2068724062 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7feeb000f040 con 0x7feea0072620 2023-10-30T20:24:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feeaffff640 10 client.4303.objecter ms_dispatch 0x7feeb8000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:24:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feeaffff640 10 client.4303.objecter _finish_command 2 = system:0 2023-10-30T20:24:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feeaffff640 20 client.4303.objecter put_session s=0x7feea4010da0 osd=0 4 2023-10-30T20:24:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feeaffff640 15 client.4303.objecter _session_command_op_remove 0 2 2023-10-30T20:24:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 10 librados: watch_flush enter 2023-10-30T20:24:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 10 librados: watch_flush exit 2023-10-30T20:24:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 10 client.4303.objecter close_session for osd.0 2023-10-30T20:24:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 1 -- 172.21.15.69:0/553467047 >> v2:172.21.15.69:6801/2068724062 conn(0x7feea0072620 msgr2=0x7feea0074a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 1 --2- 172.21.15.69:0/553467047 >> v2:172.21.15.69:6801/2068724062 conn(0x7feea0072620 0x7feea0074a40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 20 client.4303.objecter put_session s=0x7feea4010da0 osd=0 1 2023-10-30T20:24:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 20 client.4303.objecter shutdown clearing up homeless session... 2023-10-30T20:24:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 10 client.4303.objecter successfully canceled tick 2023-10-30T20:24:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 1 -- 172.21.15.69:0/553467047 >> v2:172.21.15.69:6800/1876307353 conn(0x7feea006c810 msgr2=0x7feea006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 1 --2- 172.21.15.69:0/553467047 >> v2:172.21.15.69:6800/1876307353 conn(0x7feea006c810 0x7feea006ecd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feea8035000 tx=0x7feea8030000 comp rx=0 tx=0).stop 2023-10-30T20:24:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 10 monclient: shutdown 2023-10-30T20:24:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 1 -- 172.21.15.69:0/553467047 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feeb80ab6b0 msgr2=0x7feeb813b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 1 --2- 172.21.15.69:0/553467047 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feeb80ab6b0 0x7feeb813b220 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feea400b480 tx=0x7feea400b940 comp rx=0 tx=0).stop 2023-10-30T20:24:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feeb7fff640 1 -- 172.21.15.69:0/553467047 reap_dead start 2023-10-30T20:24:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 1 -- 172.21.15.69:0/553467047 shutdown_connections 2023-10-30T20:24:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 1 -- 172.21.15.69:0/553467047 >> 172.21.15.69:0/553467047 conn(0x7feeb80b7180 msgr2=0x7feeb80b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 1 -- 172.21.15.69:0/553467047 shutdown_connections 2023-10-30T20:24:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 1 -- 172.21.15.69:0/553467047 wait complete. 2023-10-30T20:24:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.187+0000 7feebe11d640 1 librados: shutdown 2023-10-30T20:24:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.192+0000 7fae7b7fe640 1 -- 172.21.15.69:0/1377797277 <== osd.5 v2:172.21.15.183:6812/2276632263 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fae8c001a70 con 0x7fae80072620 2023-10-30T20:24:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.192+0000 7fae7b7fe640 10 client.4300.objecter ms_dispatch 0x7fae94000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:24:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.192+0000 7fae7b7fe640 10 client.4300.objecter _finish_command 2 = system:0 2023-10-30T20:24:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.192+0000 7fae7b7fe640 20 client.4300.objecter put_session s=0x7fae84010da0 osd=5 4 2023-10-30T20:24:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.192+0000 7fae7b7fe640 15 client.4300.objecter _session_command_op_remove 5 2 2023-10-30T20:24:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 10 librados: watch_flush enter 2023-10-30T20:24:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 10 librados: watch_flush exit 2023-10-30T20:24:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 10 client.4300.objecter close_session for osd.5 2023-10-30T20:24:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 1 -- 172.21.15.69:0/1377797277 >> v2:172.21.15.183:6812/2276632263 conn(0x7fae80072620 msgr2=0x7fae80074a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 1 --2- 172.21.15.69:0/1377797277 >> v2:172.21.15.183:6812/2276632263 conn(0x7fae80072620 0x7fae80074a40 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 20 client.4300.objecter put_session s=0x7fae84010da0 osd=5 1 2023-10-30T20:24:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 20 client.4300.objecter shutdown clearing up homeless session... 2023-10-30T20:24:07.449 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 10 client.4300.objecter successfully canceled tick 2023-10-30T20:24:07.449 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 1 -- 172.21.15.69:0/1377797277 >> v2:172.21.15.69:6800/1876307353 conn(0x7fae8006c810 msgr2=0x7fae8006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.449 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 1 --2- 172.21.15.69:0/1377797277 >> v2:172.21.15.69:6800/1876307353 conn(0x7fae8006c810 0x7fae8006ecd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fae88036000 tx=0x7fae8803d040 comp rx=0 tx=0).stop 2023-10-30T20:24:07.449 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 10 monclient: shutdown 2023-10-30T20:24:07.450 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.450 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 1 -- 172.21.15.69:0/1377797277 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae940a8e40 msgr2=0x7fae940b2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.450 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 1 --2- 172.21.15.69:0/1377797277 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae940a8e40 0x7fae940b2340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fae8400b4d0 tx=0x7fae8400b990 comp rx=0 tx=0).stop 2023-10-30T20:24:07.450 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae937fe640 1 -- 172.21.15.69:0/1377797277 reap_dead start 2023-10-30T20:24:07.450 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 1 -- 172.21.15.69:0/1377797277 shutdown_connections 2023-10-30T20:24:07.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 1 -- 172.21.15.69:0/1377797277 >> 172.21.15.69:0/1377797277 conn(0x7fae940b7180 msgr2=0x7fae9413ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 1 -- 172.21.15.69:0/1377797277 shutdown_connections 2023-10-30T20:24:07.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 1 -- 172.21.15.69:0/1377797277 wait complete. 2023-10-30T20:24:07.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.193+0000 7fae99621640 1 librados: shutdown 2023-10-30T20:24:07.452 INFO:teuthology.orchestra.run.smithi069.stdout:68719476743 2023-10-30T20:24:07.453 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T20:24:07.453 INFO:teuthology.orchestra.run.smithi069.stdout:64424509444 2023-10-30T20:24:07.454 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T20:24:07.455 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.283+0000 7fb99207b640 1 --2- 172.21.15.69:0/1022410054 >> v2:172.21.15.183:6800/4151811482 conn(0x7fb970072800 0x7fb970074c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.455 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.283+0000 7fb99207b640 10 monclient: get_auth_request con 0x7fb970072800 auth_method 0 2023-10-30T20:24:07.456 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.286+0000 7fb99207b640 1 --2- 172.21.15.69:0/1022410054 >> v2:172.21.15.183:6800/4151811482 conn(0x7fb970072800 0x7fb970074c20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.456 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.286+0000 7fb9827fc640 10 client.4294.objecter ms_handle_connect 0x7fb970072800 2023-10-30T20:24:07.456 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.330+0000 7fb9827fc640 1 -- 172.21.15.69:0/1022410054 <== osd.4 v2:172.21.15.183:6800/4151811482 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fb970075290 con 0x7fb970072800 2023-10-30T20:24:07.456 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.330+0000 7fb9827fc640 10 client.4294.objecter ms_dispatch 0x7fb98c000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:24:07.456 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.330+0000 7fb9827fc640 10 client.4294.objecter _finish_command 1 = system:0 2023-10-30T20:24:07.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.330+0000 7fb9827fc640 20 client.4294.objecter put_session s=0x7fb970072150 osd=4 4 2023-10-30T20:24:07.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.330+0000 7fb9827fc640 15 client.4294.objecter _session_command_op_remove 4 1 2023-10-30T20:24:07.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.343+0000 7fb993ade640 10 client.4294.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T20:24:07.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.343+0000 7fb993ade640 15 client.4294.objecter _session_command_op_assign -1 2 2023-10-30T20:24:07.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.343+0000 7fb993ade640 20 client.4294.objecter _get_session s=0x7fb970072150 osd=4 3 2023-10-30T20:24:07.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.343+0000 7fb993ade640 20 client.4294.objecter put_session s=0x7fb970072150 osd=4 3 2023-10-30T20:24:07.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.343+0000 7fb993ade640 20 client.4294.objecter _get_session s=0x7fb970072150 osd=4 3 2023-10-30T20:24:07.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.343+0000 7fb993ade640 15 client.4294.objecter _session_command_op_remove -1 2 2023-10-30T20:24:07.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.343+0000 7fb993ade640 20 client.4294.objecter get_session s=0x7fb970072150 osd=4 3 2023-10-30T20:24:07.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.343+0000 7fb993ade640 15 client.4294.objecter _session_command_op_assign 4 2 2023-10-30T20:24:07.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.343+0000 7fb993ade640 20 client.4294.objecter put_session s=0x7fb970072150 osd=4 4 2023-10-30T20:24:07.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.343+0000 7fb993ade640 10 client.4294.objecter _send_command 2 2023-10-30T20:24:07.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.343+0000 7fb993ade640 1 -- 172.21.15.69:0/1022410054 --> v2:172.21.15.183:6800/4151811482 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb98c06f320 con 0x7fb970072800 2023-10-30T20:24:07.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb9827fc640 1 -- 172.21.15.69:0/1022410054 <== osd.4 v2:172.21.15.183:6800/4151811482 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb98400f040 con 0x7fb970072800 2023-10-30T20:24:07.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb9827fc640 10 client.4294.objecter ms_dispatch 0x7fb98c000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:24:07.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb9827fc640 10 client.4294.objecter _finish_command 2 = system:0 2023-10-30T20:24:07.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb9827fc640 20 client.4294.objecter put_session s=0x7fb970072150 osd=4 4 2023-10-30T20:24:07.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb9827fc640 15 client.4294.objecter _session_command_op_remove 4 2 2023-10-30T20:24:07.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 10 librados: watch_flush enter 2023-10-30T20:24:07.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 10 librados: watch_flush exit 2023-10-30T20:24:07.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 10 client.4294.objecter close_session for osd.4 2023-10-30T20:24:07.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 1 -- 172.21.15.69:0/1022410054 >> v2:172.21.15.183:6800/4151811482 conn(0x7fb970072800 msgr2=0x7fb970074c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 1 --2- 172.21.15.69:0/1022410054 >> v2:172.21.15.183:6800/4151811482 conn(0x7fb970072800 0x7fb970074c20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 20 client.4294.objecter put_session s=0x7fb970072150 osd=4 1 2023-10-30T20:24:07.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 20 client.4294.objecter shutdown clearing up homeless session... 2023-10-30T20:24:07.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 10 client.4294.objecter successfully canceled tick 2023-10-30T20:24:07.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 1 -- 172.21.15.69:0/1022410054 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb97006c880 msgr2=0x7fb97006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 1 --2- 172.21.15.69:0/1022410054 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb97006c880 0x7fb97006ed40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb988035000 tx=0x7fb98802e000 comp rx=0 tx=0).stop 2023-10-30T20:24:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 10 monclient: shutdown 2023-10-30T20:24:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 1 -- 172.21.15.69:0/1022410054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98c13eb80 msgr2=0x7fb98c139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 1 --2- 172.21.15.69:0/1022410054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98c13eb80 0x7fb98c139f60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00d800 tx=0x7fb97c00dcc0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb99207b640 1 -- 172.21.15.69:0/1022410054 reap_dead start 2023-10-30T20:24:07.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 1 -- 172.21.15.69:0/1022410054 shutdown_connections 2023-10-30T20:24:07.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 1 -- 172.21.15.69:0/1022410054 >> 172.21.15.69:0/1022410054 conn(0x7fb98c0afba0 msgr2=0x7fb98c0ad580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 1 -- 172.21.15.69:0/1022410054 shutdown_connections 2023-10-30T20:24:07.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 1 -- 172.21.15.69:0/1022410054 wait complete. 2023-10-30T20:24:07.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.365+0000 7fb993ade640 1 librados: shutdown 2023-10-30T20:24:07.466 INFO:teuthology.orchestra.run.smithi069.stdout:55834574854 2023-10-30T20:24:07.466 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T20:24:07.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.656+0000 7f6bc3fff640 10 monclient: get_monmap_and_config 2023-10-30T20:24:07.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.656+0000 7f6bc3fff640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.656+0000 7f6bc3fff640 10 monclient: monmap: 2023-10-30T20:24:07.658 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.658 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.658 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:07.657287+0000 2023-10-30T20:24:07.659 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:07.657287+0000 2023-10-30T20:24:07.659 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.659 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.659 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:07.659 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:07.659 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:07.660 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:07.660 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:07.660 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.657+0000 7f4ec99a9640 10 monclient: get_monmap_and_config 2023-10-30T20:24:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.657+0000 7f4ec99a9640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.657+0000 7f4ec99a9640 10 monclient: monmap: 2023-10-30T20:24:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:07.658543+0000 2023-10-30T20:24:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:07.658543+0000 2023-10-30T20:24:07.662 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.662 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.662 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:07.662 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:07.662 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:07.663 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:07.663 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:07.663 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.657+0000 7f6bc3fff640 1 Processor -- start 2023-10-30T20:24:07.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.657+0000 7f6bc3fff640 1 -- start start 2023-10-30T20:24:07.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.657+0000 7f6bc3fff640 10 monclient: init 2023-10-30T20:24:07.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-10-30T20:24:07.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6bbc0070a0 0x7f6bbc0338a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f6bbc0070a0 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:07.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bbc0307d0 0x7f6bbc035b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f6bbc0307d0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6bbc0360b0 0x7f6bbc038db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f6bbc0360b0 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:07.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bbc008610 con 0x7f6bbc0070a0 2023-10-30T20:24:07.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bbc007480 con 0x7f6bbc0360b0 2023-10-30T20:24:07.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0375f0 con 0x7f6bbc0307d0 2023-10-30T20:24:07.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.658+0000 7f6bc3fff640 10 monclient(hunting): authenticate will time out at 1570.947713s 2023-10-30T20:24:07.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.662+0000 7f4ec99a9640 1 Processor -- start 2023-10-30T20:24:07.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.662+0000 7f4ec99a9640 1 -- start start 2023-10-30T20:24:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.662+0000 7f4ec99a9640 10 monclient: init 2023-10-30T20:24:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.662+0000 7f4ec99a9640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.663+0000 7f4ec99a9640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-10-30T20:24:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.663+0000 7f4ec99a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ec40a88d0 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.663+0000 7f4ec99a9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4ec40b1b00 0x7f4ec40a8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.663+0000 7f4ec99a9640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec40b1b00 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.663+0000 7f4ec99a9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec4143220 0x7f4ec4143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.663+0000 7f4ec99a9640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec4143220 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.663+0000 7f4ec99a9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec40a91f0 con 0x7f4ec40a88d0 2023-10-30T20:24:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.663+0000 7f4ec99a9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.663+0000 7f4ec99a9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec40679a0 con 0x7f4ec40b1b00 2023-10-30T20:24:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.663+0000 7f4ec99a9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.663+0000 7f4ec99a9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec4147070 con 0x7f4ec4143220 2023-10-30T20:24:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.663+0000 7f4ec99a9640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.663+0000 7f4ec99a9640 10 monclient(hunting): authenticate will time out at 1570.951933s 2023-10-30T20:24:07.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.669+0000 7f6bc27fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bbc0307d0 0x7f6bbc035b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.669+0000 7f6bc27fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bbc0307d0 0x7f6bbc035b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60728/0 (socket says 172.21.15.69:60728) 2023-10-30T20:24:07.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.669+0000 7f6bc27fc640 1 -- 172.21.15.69:0/860255227 learned_addr learned my addr 172.21.15.69:0/860255227 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.669+0000 7f6bc27fc640 10 monclient(hunting): get_auth_request con 0x7f6bbc0307d0 auth_method 0 2023-10-30T20:24:07.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.669+0000 7f6bc27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.669+0000 7f6bc27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.669+0000 7f6bc27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.666+0000 7fa63dba9640 10 monclient: get_monmap_and_config 2023-10-30T20:24:07.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.666+0000 7fa63dba9640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.666+0000 7fa63dba9640 10 monclient: monmap: 2023-10-30T20:24:07.677 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.678 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.678 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:07.667695+0000 2023-10-30T20:24:07.678 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:07.667695+0000 2023-10-30T20:24:07.678 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.678 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.679 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:07.679 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:07.679 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:07.679 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:07.679 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:07.680 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.680 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.667+0000 7fa63dba9640 1 Processor -- start 2023-10-30T20:24:07.680 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.667+0000 7fa63dba9640 1 -- start start 2023-10-30T20:24:07.680 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.667+0000 7fa63dba9640 10 monclient: init 2023-10-30T20:24:07.680 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-10-30T20:24:07.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa638151df0 0x7fa63806ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 10 monclient(hunting): picked mon.noname-a con 0x7fa638151df0 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:07.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa6381521d0 0x7fa638150e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 10 monclient(hunting): picked mon.noname-c con 0x7fa6381521d0 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:07.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6381513b0 0x7fa6380ad680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 10 monclient(hunting): picked mon.noname-b con 0x7fa6381513b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa638067b90 con 0x7fa638151df0 2023-10-30T20:24:07.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa63806e2c0 con 0x7fa6381521d0 2023-10-30T20:24:07.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63806fdc0 con 0x7fa6381513b0 2023-10-30T20:24:07.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.668+0000 7fa63dba9640 10 monclient(hunting): authenticate will time out at 1570.957696s 2023-10-30T20:24:07.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.670+0000 7fa637fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6381513b0 0x7fa6380ad680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.670+0000 7fa637fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6381513b0 0x7fa6380ad680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60738/0 (socket says 172.21.15.69:60738) 2023-10-30T20:24:07.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.670+0000 7fa637fff640 1 -- 172.21.15.69:0/1333403722 learned_addr learned my addr 172.21.15.69:0/1333403722 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.670+0000 7fa637fff640 10 monclient(hunting): get_auth_request con 0x7fa6381513b0 auth_method 0 2023-10-30T20:24:07.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.670+0000 7fa637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.670+0000 7fa637fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.670+0000 7fa637fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.671+0000 7fa637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.671+0000 7fa637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.671+0000 7fa637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.672+0000 7f798cd67640 10 monclient: get_monmap_and_config 2023-10-30T20:24:07.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.672+0000 7f798cd67640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.672+0000 7f798cd67640 10 monclient: monmap: 2023-10-30T20:24:07.686 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.687 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.687 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:07.673040+0000 2023-10-30T20:24:07.687 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:07.673040+0000 2023-10-30T20:24:07.687 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.687 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.687 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:07.688 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:07.688 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:07.688 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:07.688 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:07.688 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.673+0000 7f6bc37fe640 1 --2- 172.21.15.69:0/860255227 >> v2:172.21.15.183:3300/0 conn(0x7f6bbc0360b0 0x7f6bbc038db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.673+0000 7fe345fe9640 10 monclient: get_monmap_and_config 2023-10-30T20:24:07.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.673+0000 7fe345fe9640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.673+0000 7fe345fe9640 10 monclient: monmap: 2023-10-30T20:24:07.690 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.690 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.690 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:07.674206+0000 2023-10-30T20:24:07.690 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:07.674206+0000 2023-10-30T20:24:07.690 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.691 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.691 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:07.691 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:07.691 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:07.691 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:07.691 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:07.692 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.673+0000 7f798cd67640 1 Processor -- start 2023-10-30T20:24:07.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.673+0000 7f4ec27fc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4ec40b1b00 0x7f4ec40a8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.673+0000 7f798cd67640 1 -- start start 2023-10-30T20:24:07.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.673+0000 7f798cd67640 10 monclient: init 2023-10-30T20:24:07.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f4ec27fc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4ec40b1b00 0x7f4ec40a8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36516/0 (socket says 172.21.15.69:36516) 2023-10-30T20:24:07.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f4ec27fc640 1 -- 172.21.15.69:0/25291156 learned_addr learned my addr 172.21.15.69:0/25291156 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f4ec27fc640 10 monclient(hunting): get_auth_request con 0x7f4ec40b1b00 auth_method 0 2023-10-30T20:24:07.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f4ec27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f4ec27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f4ec27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f6bc2ffd640 1 --2- 172.21.15.69:0/860255227 >> v2:172.21.15.69:3300/0 conn(0x7f6bbc0070a0 0x7f6bbc0338a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7fe345fe9640 1 Processor -- start 2023-10-30T20:24:07.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f798cd67640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f798cd67640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2023-10-30T20:24:07.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f798cd67640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7980034aa0 addr v1:172.21.15.69:6789/0 2023-10-30T20:24:07.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f798cd67640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7980034e80 0x7f798002a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f798cd67640 10 monclient(hunting): picked mon.noname-a con 0x7f7980034e80 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:07.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f798cd67640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f79800c3d00 0x7f79800c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f798cd67640 10 monclient(hunting): picked mon.noname-c con 0x7f79800c3d00 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:07.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f798cd67640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7980032f60 con 0x7f7980034aa0 2023-10-30T20:24:07.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f798cd67640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f798cd67640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7980008720 con 0x7f7980034e80 2023-10-30T20:24:07.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f798cd67640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f798cd67640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f798002cb10 con 0x7f79800c3d00 2023-10-30T20:24:07.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f798cd67640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7f798cd67640 10 monclient(hunting): authenticate will time out at 1570.963336s 2023-10-30T20:24:07.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7fe345fe9640 1 -- start start 2023-10-30T20:24:07.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7fe345fe9640 10 monclient: init 2023-10-30T20:24:07.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7fa6377fe640 1 --2- 172.21.15.69:0/1333403722 >> v2:172.21.15.69:3300/0 conn(0x7fa638151df0 0x7fa63806ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.674+0000 7fa636ffd640 1 --2- 172.21.15.69:0/1333403722 >> v2:172.21.15.183:3300/0 conn(0x7fa6381521d0 0x7fa638150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7f6bc37fe640 10 monclient(hunting): get_auth_request con 0x7f6bbc0360b0 auth_method 0 2023-10-30T20:24:07.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7f6bc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7f6bc37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7f6bc37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7f4ec27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7f4ec27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7f4ec27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fa636ffd640 10 monclient(hunting): get_auth_request con 0x7fa6381521d0 auth_method 0 2023-10-30T20:24:07.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fa636ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fa636ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fa636ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fe345fe9640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fe345fe9640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-10-30T20:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fe345fe9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe3400a8e40 0x7fe34013efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fe345fe9640 10 monclient(hunting): picked mon.noname-c con 0x7fe3400a8e40 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fe345fe9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3400b13c0 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fe345fe9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe34013f500 0x7fe3401425c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fe345fe9640 10 monclient(hunting): picked mon.noname-a con 0x7fe34013f500 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:07.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fe345fe9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3400b1b00 con 0x7fe3400b13c0 2023-10-30T20:24:07.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fe345fe9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fe345fe9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3400679a0 con 0x7fe34013f500 2023-10-30T20:24:07.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fe345fe9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fe345fe9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3400b1060 con 0x7fe3400a8e40 2023-10-30T20:24:07.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fe345fe9640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.675+0000 7fe345fe9640 10 monclient(hunting): authenticate will time out at 1570.964754s 2023-10-30T20:24:07.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.676+0000 7f4ec27fc640 10 monclient(hunting): handle_auth_done global_id 4306 payload 274 2023-10-30T20:24:07.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.676+0000 7f4ec27fc640 1 -- 172.21.15.69:0/25291156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec4143220 msgr2=0x7f4ec4143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.676+0000 7f4ec27fc640 1 --2- 172.21.15.69:0/25291156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec4143220 0x7f4ec4143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.676+0000 7f4ec27fc640 1 -- 172.21.15.69:0/25291156 >> v1:172.21.15.183:6789/0 conn(0x7f4ec40a88d0 legacy=0x7f4ec4146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.676+0000 7f4ec27fc640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.676+0000 7f4ec27fc640 1 monclient: found mon.noname-c 2023-10-30T20:24:07.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.677+0000 7f4ec27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.677+0000 7f4ec27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.677+0000 7f4ec27fc640 1 -- 172.21.15.69:0/25291156 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec40aa250 con 0x7f4ec40b1b00 2023-10-30T20:24:07.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.678+0000 7fa636ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.678+0000 7fa636ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.678+0000 7fa636ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.679+0000 7f7fbd390640 10 monclient: get_monmap_and_config 2023-10-30T20:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.679+0000 7f7fbd390640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.679+0000 7f7fbd390640 10 monclient: monmap: 2023-10-30T20:24:07.712 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.712 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.712 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:07.680016+0000 2023-10-30T20:24:07.712 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:07.680016+0000 2023-10-30T20:24:07.712 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.713 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.713 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:07.713 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:07.713 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:07.713 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:07.714 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:07.714 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.679+0000 7f7fbd390640 1 Processor -- start 2023-10-30T20:24:07.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.679+0000 7f7fbd390640 1 -- start start 2023-10-30T20:24:07.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.679+0000 7f7fbd390640 10 monclient: init 2023-10-30T20:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.680+0000 7f7fbd390640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.680+0000 7f7fbd390640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2023-10-30T20:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.680+0000 7f7fbd390640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb80a8e40 0x7f7fb813efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.680+0000 7f7fbd390640 10 monclient(hunting): picked mon.noname-b con 0x7f7fb80a8e40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.680+0000 7f7fbd390640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7fb80b13c0 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.680+0000 7f7fbd390640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7fb813f500 0x7f7fb81425c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.680+0000 7f7fbd390640 10 monclient(hunting): picked mon.noname-a con 0x7f7fb813f500 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.680+0000 7f7fbd390640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb80b1b00 con 0x7f7fb80b13c0 2023-10-30T20:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.680+0000 7f7fbd390640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.680+0000 7f7fbd390640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb80679a0 con 0x7f7fb813f500 2023-10-30T20:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.680+0000 7f7fbd390640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.680+0000 7f7fbd390640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb80b1060 con 0x7f7fb80a8e40 2023-10-30T20:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.680+0000 7f7fbd390640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.680+0000 7f7fbd390640 10 monclient(hunting): authenticate will time out at 1570.969314s 2023-10-30T20:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.681+0000 7fa636ffd640 10 monclient(hunting): handle_auth_done global_id 4309 payload 274 2023-10-30T20:24:07.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.681+0000 7fa636ffd640 1 -- 172.21.15.69:0/1333403722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6381513b0 msgr2=0x7fa6380ad680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.681+0000 7fa636ffd640 1 --2- 172.21.15.69:0/1333403722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6381513b0 0x7fa6380ad680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.681+0000 7fa636ffd640 1 -- 172.21.15.69:0/1333403722 >> v2:172.21.15.69:3300/0 conn(0x7fa638151df0 msgr2=0x7fa63806ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.681+0000 7fa636ffd640 1 --2- 172.21.15.69:0/1333403722 >> v2:172.21.15.69:3300/0 conn(0x7fa638151df0 0x7fa63806ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.681+0000 7fa636ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.681+0000 7fa636ffd640 1 monclient: found mon.noname-c 2023-10-30T20:24:07.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.681+0000 7fa636ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.681+0000 7fa636ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.681+0000 7fa636ffd640 1 -- 172.21.15.69:0/1333403722 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6380700a0 con 0x7fa6381521d0 2023-10-30T20:24:07.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.682+0000 7f75149ed640 10 monclient: get_monmap_and_config 2023-10-30T20:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.682+0000 7f75149ed640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.682+0000 7f75149ed640 10 monclient: monmap: 2023-10-30T20:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:07.683174+0000 2023-10-30T20:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:07.683174+0000 2023-10-30T20:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.682+0000 7f75149ed640 1 Processor -- start 2023-10-30T20:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.682+0000 7f75149ed640 1 -- start start 2023-10-30T20:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.682+0000 7f75149ed640 10 monclient: init 2023-10-30T20:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-10-30T20:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7508037670 0x7f75080c1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 10 monclient(hunting): picked mon.noname-b con 0x7f7508037670 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f750802c6d0 0x7f75080c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 10 monclient(hunting): picked mon.noname-c con 0x7f750802c6d0 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f750802b200 0x7f75080c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 10 monclient(hunting): picked mon.noname-a con 0x7f750802b200 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7508009840 con 0x7f750802b200 2023-10-30T20:24:07.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7508037a50 con 0x7f750802c6d0 2023-10-30T20:24:07.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750802c280 con 0x7f7508037670 2023-10-30T20:24:07.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.683+0000 7f75149ed640 10 monclient(hunting): authenticate will time out at 1570.972585s 2023-10-30T20:24:07.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.684+0000 7fa636ffd640 1 --2- 172.21.15.69:0/1333403722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6381521d0 0x7fa638150e70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa6380af2c0 tx=0x7fa62c02b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5362382cdd24a023 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.684+0000 7f4ec27fc640 1 --2- 172.21.15.69:0/25291156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ec40b1b00 0x7f4ec40a8cb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4ec406bdf0 tx=0x7f4ebc02be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80e3e788e15e16b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.688+0000 7f6bc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.688+0000 7f6bc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.688+0000 7f6bc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.690+0000 7f750e59c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f750802c6d0 0x7f75080c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.691+0000 7f750e59c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f750802c6d0 0x7f75080c2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36548/0 (socket says 172.21.15.69:36548) 2023-10-30T20:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.691+0000 7f750e59c640 1 -- 172.21.15.69:0/2976867836 learned_addr learned my addr 172.21.15.69:0/2976867836 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.691+0000 7f750e59c640 10 monclient(hunting): get_auth_request con 0x7f750802c6d0 auth_method 0 2023-10-30T20:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.691+0000 7f750e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.691+0000 7f750e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.691+0000 7f750e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.691+0000 7fe33f7fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe3400a8e40 0x7fe34013efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.691+0000 7f7987fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f79800c3d00 0x7f79800c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7f750e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7f750e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7f750e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7fe33f7fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe3400a8e40 0x7fe34013efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36546/0 (socket says 172.21.15.69:36546) 2023-10-30T20:24:07.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7fe33f7fe640 1 -- 172.21.15.69:0/1516023723 learned_addr learned my addr 172.21.15.69:0/1516023723 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7fe33f7fe640 10 monclient(hunting): get_auth_request con 0x7fe3400a8e40 auth_method 0 2023-10-30T20:24:07.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7fe33f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7fe33f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7fe33f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7f7987fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f79800c3d00 0x7f79800c4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36530/0 (socket says 172.21.15.69:36530) 2023-10-30T20:24:07.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7f7987fff640 1 -- 172.21.15.69:0/3531338428 learned_addr learned my addr 172.21.15.69:0/3531338428 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7f7987fff640 10 monclient(hunting): get_auth_request con 0x7f79800c3d00 auth_method 0 2023-10-30T20:24:07.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7f7987fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7f7987fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.692+0000 7f7987fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.693+0000 7f750e59c640 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2023-10-30T20:24:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.693+0000 7f750e59c640 1 -- 172.21.15.69:0/2976867836 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7508037670 msgr2=0x7f75080c1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.693+0000 7f750e59c640 1 --2- 172.21.15.69:0/2976867836 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7508037670 0x7f75080c1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.693+0000 7f750e59c640 1 -- 172.21.15.69:0/2976867836 >> v2:172.21.15.69:3300/0 conn(0x7f750802b200 msgr2=0x7f75080c7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.693+0000 7f750e59c640 1 --2- 172.21.15.69:0/2976867836 >> v2:172.21.15.69:3300/0 conn(0x7f750802b200 0x7f75080c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.693+0000 7f750e59c640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.693+0000 7f750e59c640 1 monclient: found mon.noname-c 2023-10-30T20:24:07.737 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.693+0000 7f750e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.737 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.693+0000 7f750e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.737 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.693+0000 7f750e59c640 1 -- 172.21.15.69:0/2976867836 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750802b7b0 con 0x7f750802c6d0 2023-10-30T20:24:07.737 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.695+0000 7f7987fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.695+0000 7f7987fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.695+0000 7f7987fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.696+0000 7f7987fff640 10 monclient(hunting): handle_auth_done global_id 4318 payload 274 2023-10-30T20:24:07.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.696+0000 7f7987fff640 1 -- 172.21.15.69:0/3531338428 >> v2:172.21.15.69:3300/0 conn(0x7f7980034e80 msgr2=0x7f798002a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.696+0000 7f7987fff640 1 --2- 172.21.15.69:0/3531338428 >> v2:172.21.15.69:3300/0 conn(0x7f7980034e80 0x7f798002a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.696+0000 7f6bc2ffd640 10 monclient(hunting): get_auth_request con 0x7f6bbc0070a0 auth_method 0 2023-10-30T20:24:07.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.696+0000 7f6bc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.696+0000 7f6bc2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.696+0000 7f6bc2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.740 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.696+0000 7f7987fff640 1 -- 172.21.15.69:0/3531338428 >> v1:172.21.15.69:6789/0 conn(0x7f7980034aa0 legacy=0x7f7980032850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.741 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.696+0000 7f7987fff640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.741 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.697+0000 7f7987fff640 1 monclient: found mon.noname-c 2023-10-30T20:24:07.741 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.697+0000 7f7987fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.741 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.697+0000 7f7987fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.741 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.697+0000 7f7987fff640 1 -- 172.21.15.69:0/3531338428 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798002cd10 con 0x7f79800c3d00 2023-10-30T20:24:07.742 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.697+0000 7fe33f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.742 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.697+0000 7fe33f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.742 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.697+0000 7fe33f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.697+0000 7f6bc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.698+0000 7f6bc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.698+0000 7f6bc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.698+0000 7f750e59c640 1 --2- 172.21.15.69:0/2976867836 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f750802c6d0 0x7f75080c2120 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7508037d70 tx=0x7f750402b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ac293724db4a0e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.744 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.700+0000 7f6bc2ffd640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2023-10-30T20:24:07.744 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.700+0000 7f6bc2ffd640 1 -- 172.21.15.69:0/860255227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bbc0307d0 msgr2=0x7f6bbc035b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.744 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.700+0000 7f6bc2ffd640 1 --2- 172.21.15.69:0/860255227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bbc0307d0 0x7f6bbc035b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.744 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.700+0000 7f6bc2ffd640 1 -- 172.21.15.69:0/860255227 >> v2:172.21.15.183:3300/0 conn(0x7f6bbc0360b0 msgr2=0x7f6bbc038db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.745 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.700+0000 7f6bc2ffd640 1 --2- 172.21.15.69:0/860255227 >> v2:172.21.15.183:3300/0 conn(0x7f6bbc0360b0 0x7f6bbc038db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.745 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.700+0000 7f6bc2ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.745 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.700+0000 7f6bc2ffd640 1 monclient: found mon.noname-a 2023-10-30T20:24:07.745 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.700+0000 7f6bc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.745 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.700+0000 7f6bc2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.700+0000 7f6bc2ffd640 1 -- 172.21.15.69:0/860255227 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc036540 con 0x7f6bbc0070a0 2023-10-30T20:24:07.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa6367fc640 1 -- 172.21.15.69:0/1333403722 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c02e050 con 0x7fa6381521d0 2023-10-30T20:24:07.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa6367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa6367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:07.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa6367fc640 10 monclient: dump: 2023-10-30T20:24:07.748 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.748 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.748 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.748 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.749 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.749 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.749 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.749 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.749 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.750 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.750 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa6367fc640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.750 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa6367fc640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.750 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa6367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.750 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa6367fc640 1 -- 172.21.15.69:0/1333403722 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62c030710 con 0x7fa6381521d0 2023-10-30T20:24:07.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa6367fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa6367fc640 1 -- 172.21.15.69:0/1333403722 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c030ab0 con 0x7fa6381521d0 2023-10-30T20:24:07.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa6367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa6367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa6367fc640 10 monclient: dump: 2023-10-30T20:24:07.752 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.752 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.752 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.752 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.752 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.752 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.753 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.753 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.753 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.753 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.753 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa63dba9640 5 monclient: authenticate success, global_id 4309 2023-10-30T20:24:07.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.701+0000 7fa63dba9640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:07.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.702+0000 7fa63dba9640 4 set_mon_vals no callback set 2023-10-30T20:24:07.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.702+0000 7fa63dba9640 10 monclient: shutdown 2023-10-30T20:24:07.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.702+0000 7fa63dba9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.702+0000 7fa63dba9640 1 -- 172.21.15.69:0/1333403722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6381521d0 msgr2=0x7fa638150e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.702+0000 7fa63dba9640 1 --2- 172.21.15.69:0/1333403722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6381521d0 0x7fa638150e70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa6380af2c0 tx=0x7fa62c02b710 comp rx=0 tx=0).stop 2023-10-30T20:24:07.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.702+0000 7fa63dba9640 1 -- 172.21.15.69:0/1333403722 shutdown_connections 2023-10-30T20:24:07.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.702+0000 7fa63dba9640 1 --2- 172.21.15.69:0/1333403722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6381513b0 0x7fa6380ad680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.702+0000 7fa63dba9640 1 --2- 172.21.15.69:0/1333403722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6381521d0 0x7fa638150e70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.702+0000 7fa63dba9640 1 --2- 172.21.15.69:0/1333403722 >> v2:172.21.15.69:3300/0 conn(0x7fa638151df0 0x7fa63806ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.702+0000 7fa63dba9640 1 -- 172.21.15.69:0/1333403722 >> 172.21.15.69:0/1333403722 conn(0x7fa6380afba0 msgr2=0x7fa6380affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.703+0000 7fa63dba9640 1 -- 172.21.15.69:0/1333403722 shutdown_connections 2023-10-30T20:24:07.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.703+0000 7fa63dba9640 1 -- 172.21.15.69:0/1333403722 wait complete. 2023-10-30T20:24:07.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.703+0000 7fa63dba9640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.703+0000 7fa63dba9640 10 monclient: monmap: 2023-10-30T20:24:07.756 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.757 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.757 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.757 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.757 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.757 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.758 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.758 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.758 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.758 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.703+0000 7f6bc2ffd640 1 --2- 172.21.15.69:0/860255227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bbc0070a0 0x7f6bbc0338a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc010060 tx=0x7f6bb802b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ddf1d2fce89d09 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.703+0000 7fa63dba9640 1 librados: starting msgr at 2023-10-30T20:24:07.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.703+0000 7fa63dba9640 1 librados: starting objecter 2023-10-30T20:24:07.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 1 Processor -- start 2023-10-30T20:24:07.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 1 -- start start 2023-10-30T20:24:07.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 1 librados: setting wanted keys 2023-10-30T20:24:07.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 1 librados: calling monclient init 2023-10-30T20:24:07.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 10 monclient: init 2023-10-30T20:24:07.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T20:24:07.762 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6381513b0 0x7fa6380aae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.762 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 10 monclient(hunting): picked mon.noname-a con 0x7fa6381513b0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.762 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa638151df0 0x7fa63813de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.762 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 10 monclient(hunting): picked mon.noname-c con 0x7fa638151df0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.762 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6381521d0 0x7fa63813d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.763 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 10 monclient(hunting): picked mon.noname-b con 0x7fa6381521d0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.763 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.763 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6380aff80 con 0x7fa6381513b0 2023-10-30T20:24:07.763 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.763 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6381e3420 con 0x7fa6381521d0 2023-10-30T20:24:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6381e3670 con 0x7fa638151df0 2023-10-30T20:24:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.704+0000 7fa63dba9640 10 monclient(hunting): authenticate will time out at 1570.993607s 2023-10-30T20:24:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.706+0000 7f7fb77fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7fb813f500 0x7f7fb81425c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.765 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.706+0000 7fe33f7fe640 10 monclient(hunting): handle_auth_done global_id 4321 payload 274 2023-10-30T20:24:07.765 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.706+0000 7fe33f7fe640 1 -- 172.21.15.69:0/1516023723 >> v2:172.21.15.69:3300/0 conn(0x7fe34013f500 msgr2=0x7fe3401425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.706+0000 7fe33f7fe640 1 --2- 172.21.15.69:0/1516023723 >> v2:172.21.15.69:3300/0 conn(0x7fe34013f500 0x7fe3401425c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.706+0000 7fe33f7fe640 1 -- 172.21.15.69:0/1516023723 >> v1:172.21.15.183:6789/0 conn(0x7fe3400b13c0 legacy=0x7fe340141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.706+0000 7fe33f7fe640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.706+0000 7fe33f7fe640 1 monclient: found mon.noname-c 2023-10-30T20:24:07.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.706+0000 7fe33f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.706+0000 7fe33f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.706+0000 7fe33f7fe640 1 -- 172.21.15.69:0/1516023723 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3400a8920 con 0x7fe3400a8e40 2023-10-30T20:24:07.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f750dd9b640 1 -- 172.21.15.69:0/2976867836 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7504030800 con 0x7f750802c6d0 2023-10-30T20:24:07.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f750dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f750dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:07.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f750dd9b640 10 monclient: dump: 2023-10-30T20:24:07.769 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.769 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.769 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.770 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.770 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.770 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.770 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.770 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.771 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.771 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.771 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f750dd9b640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.771 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f750dd9b640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.771 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f750dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.772 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f750dd9b640 1 -- 172.21.15.69:0/2976867836 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7504030970 con 0x7f750802c6d0 2023-10-30T20:24:07.772 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f750dd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.772 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f750dd9b640 1 -- 172.21.15.69:0/2976867836 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7504030d10 con 0x7f750802c6d0 2023-10-30T20:24:07.772 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f750dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.772 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f750dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f750dd9b640 10 monclient: dump: 2023-10-30T20:24:07.773 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.773 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.773 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.773 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.774 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.774 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.774 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.774 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.774 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.775 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f75149ed640 5 monclient: authenticate success, global_id 4315 2023-10-30T20:24:07.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f75149ed640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:07.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f75149ed640 4 set_mon_vals no callback set 2023-10-30T20:24:07.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.707+0000 7f75149ed640 10 monclient: shutdown 2023-10-30T20:24:07.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f75149ed640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f75149ed640 1 -- 172.21.15.69:0/2976867836 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f750802c6d0 msgr2=0x7f75080c2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f75149ed640 1 --2- 172.21.15.69:0/2976867836 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f750802c6d0 0x7f75080c2120 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7508037d70 tx=0x7f750402b940 comp rx=0 tx=0).stop 2023-10-30T20:24:07.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f75149ed640 1 -- 172.21.15.69:0/2976867836 shutdown_connections 2023-10-30T20:24:07.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f75149ed640 1 --2- 172.21.15.69:0/2976867836 >> v2:172.21.15.69:3300/0 conn(0x7f750802b200 0x7f75080c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f75149ed640 1 --2- 172.21.15.69:0/2976867836 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f750802c6d0 0x7f75080c2120 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f75149ed640 1 --2- 172.21.15.69:0/2976867836 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7508037670 0x7f75080c1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f75149ed640 1 -- 172.21.15.69:0/2976867836 >> 172.21.15.69:0/2976867836 conn(0x7f750803a9e0 msgr2=0x7f7508036cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f75149ed640 1 -- 172.21.15.69:0/2976867836 shutdown_connections 2023-10-30T20:24:07.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f75149ed640 1 -- 172.21.15.69:0/2976867836 wait complete. 2023-10-30T20:24:07.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f7fb77fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7fb813f500 0x7f7fb81425c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45354/0 (socket says 172.21.15.69:45354) 2023-10-30T20:24:07.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f7fb77fe640 1 -- 172.21.15.69:0/597110672 learned_addr learned my addr 172.21.15.69:0/597110672 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f7fb77fe640 10 monclient(hunting): get_auth_request con 0x7f7fb813f500 auth_method 0 2023-10-30T20:24:07.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f7fb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f7fb77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.708+0000 7f7fb77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.709+0000 7f75149ed640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.709+0000 7f75149ed640 10 monclient: monmap: 2023-10-30T20:24:07.781 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.781 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.781 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.781 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.781 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.782 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.782 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.782 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.782 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.782 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.709+0000 7f75149ed640 1 librados: starting msgr at 2023-10-30T20:24:07.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.709+0000 7f75149ed640 1 librados: starting objecter 2023-10-30T20:24:07.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.709+0000 7f75149ed640 1 Processor -- start 2023-10-30T20:24:07.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.709+0000 7f75149ed640 1 -- start start 2023-10-30T20:24:07.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.709+0000 7f75149ed640 1 librados: setting wanted keys 2023-10-30T20:24:07.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.709+0000 7f75149ed640 1 librados: calling monclient init 2023-10-30T20:24:07.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.709+0000 7f75149ed640 10 monclient: init 2023-10-30T20:24:07.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T20:24:07.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750802b200 0x7f750802f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 10 monclient(hunting): picked mon.noname-c con 0x7f750802b200 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7508037670 0x7f750802f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 10 monclient(hunting): picked mon.noname-a con 0x7f7508037670 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f750815b930 0x7f750815bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 10 monclient(hunting): picked mon.noname-b con 0x7f750815b930 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7508005d30 con 0x7f7508037670 2023-10-30T20:24:07.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7508030340 con 0x7f750815b930 2023-10-30T20:24:07.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec1ffb640 1 -- 172.21.15.69:0/25291156 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc02c9a0 con 0x7f4ec40b1b00 2023-10-30T20:24:07.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75080c6c90 con 0x7f750802b200 2023-10-30T20:24:07.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f75149ed640 10 monclient(hunting): authenticate will time out at 1570.999397s 2023-10-30T20:24:07.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:07.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec1ffb640 10 monclient: dump: 2023-10-30T20:24:07.789 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.789 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.789 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.790 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.790 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.790 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.790 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.790 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.790 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.791 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec1ffb640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec1ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec99a9640 5 monclient: authenticate success, global_id 4306 2023-10-30T20:24:07.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec99a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:07.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec1ffb640 1 -- 172.21.15.69:0/25291156 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc02cb10 con 0x7f4ec40b1b00 2023-10-30T20:24:07.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec99a9640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:07.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec99a9640 4 set_mon_vals no callback set 2023-10-30T20:24:07.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec1ffb640 1 -- 172.21.15.69:0/25291156 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc0343f0 con 0x7f4ec40b1b00 2023-10-30T20:24:07.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.710+0000 7f4ec1ffb640 10 monclient: dump: 2023-10-30T20:24:07.793 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.793 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.793 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.794 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.794 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.794 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.794 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.794 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.794 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.795 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.711+0000 7f4ec99a9640 10 monclient: shutdown 2023-10-30T20:24:07.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.711+0000 7f4ec99a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.711+0000 7f4ec99a9640 1 -- 172.21.15.69:0/25291156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ec40b1b00 msgr2=0x7f4ec40a8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.711+0000 7f4ec99a9640 1 --2- 172.21.15.69:0/25291156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ec40b1b00 0x7f4ec40a8cb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4ec406bdf0 tx=0x7f4ebc02be00 comp rx=0 tx=0).stop 2023-10-30T20:24:07.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.711+0000 7f4ec99a9640 1 -- 172.21.15.69:0/25291156 shutdown_connections 2023-10-30T20:24:07.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.711+0000 7f4ec99a9640 1 --2- 172.21.15.69:0/25291156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec4143220 0x7f4ec4143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.711+0000 7f4ec99a9640 1 --2- 172.21.15.69:0/25291156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ec40b1b00 0x7f4ec40a8cb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.711+0000 7f4ec99a9640 1 -- 172.21.15.69:0/25291156 >> 172.21.15.69:0/25291156 conn(0x7f4ec40b7180 msgr2=0x7f4ec40b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.711+0000 7f4ec99a9640 1 -- 172.21.15.69:0/25291156 shutdown_connections 2023-10-30T20:24:07.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.711+0000 7f4ec99a9640 1 -- 172.21.15.69:0/25291156 wait complete. 2023-10-30T20:24:07.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.712+0000 7f7fb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.712+0000 7f7fb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.712+0000 7f7fb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.712+0000 7f4ec99a9640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.799 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.712+0000 7f4ec99a9640 10 monclient: monmap: 2023-10-30T20:24:07.799 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.799 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.799 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.799 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.800 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.800 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.800 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.800 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.800 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.800 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.801 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.712+0000 7f4ec99a9640 1 librados: starting msgr at 2023-10-30T20:24:07.801 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.712+0000 7f4ec99a9640 1 librados: starting objecter 2023-10-30T20:24:07.801 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.712+0000 7f4ec99a9640 1 Processor -- start 2023-10-30T20:24:07.801 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.712+0000 7f4ec99a9640 1 -- start start 2023-10-30T20:24:07.801 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.712+0000 7f4ec99a9640 1 librados: setting wanted keys 2023-10-30T20:24:07.801 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.712+0000 7f4ec99a9640 1 librados: calling monclient init 2023-10-30T20:24:07.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.712+0000 7f4ec99a9640 10 monclient: init 2023-10-30T20:24:07.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:24:07.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec40a88d0 0x7f4ec40ae250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 10 monclient(hunting): picked mon.noname-a con 0x7f4ec40a88d0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.803 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ec40b1b00 0x7f4ec412f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.803 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec40b1b00 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.803 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec4143220 0x7f4ec40b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.803 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec4143220 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.803 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.803 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec40a8ec0 con 0x7f4ec40a88d0 2023-10-30T20:24:07.804 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.804 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7fa637fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6381521d0 0x7fa63813d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.804 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec40b01f0 con 0x7f4ec40b1b00 2023-10-30T20:24:07.805 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.805 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec40ad6d0 con 0x7f4ec4143220 2023-10-30T20:24:07.805 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.805 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.713+0000 7f4ec99a9640 10 monclient(hunting): authenticate will time out at 1571.002214s 2023-10-30T20:24:07.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7f7fb77fe640 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2023-10-30T20:24:07.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7f7fb77fe640 1 -- 172.21.15.69:0/597110672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb80a8e40 msgr2=0x7f7fb813efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7f7fb77fe640 1 --2- 172.21.15.69:0/597110672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb80a8e40 0x7f7fb813efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7f7fb77fe640 1 -- 172.21.15.69:0/597110672 >> v1:172.21.15.183:6789/0 conn(0x7f7fb80b13c0 legacy=0x7f7fb8141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7f7fb77fe640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7f7fb77fe640 1 monclient: found mon.noname-a 2023-10-30T20:24:07.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7f7fb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7f7fb77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7f7fb77fe640 1 -- 172.21.15.69:0/597110672 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb80a8920 con 0x7f7fb813f500 2023-10-30T20:24:07.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7fa637fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6381521d0 0x7fa63813d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36554/0 (socket says 172.21.15.69:36554) 2023-10-30T20:24:07.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7fa637fff640 1 -- 172.21.15.69:0/3426682588 learned_addr learned my addr 172.21.15.69:0/3426682588 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7fa637fff640 10 monclient(hunting): get_auth_request con 0x7fa6381521d0 auth_method 0 2023-10-30T20:24:07.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7fa637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7fa637fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.715+0000 7fa637fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.716+0000 7f750f59e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f750815b930 0x7f750815bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.717+0000 7f14953d5640 10 monclient: get_monmap_and_config 2023-10-30T20:24:07.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.717+0000 7f14953d5640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.718+0000 7f14953d5640 10 monclient: monmap: 2023-10-30T20:24:07.811 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.811 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.811 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:07.718962+0000 2023-10-30T20:24:07.811 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:07.718962+0000 2023-10-30T20:24:07.811 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.812 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.812 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:07.812 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:07.812 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:07.812 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:07.813 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:07.813 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.813 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.718+0000 7f14953d5640 1 Processor -- start 2023-10-30T20:24:07.813 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.718+0000 7f14953d5640 1 -- start start 2023-10-30T20:24:07.813 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.718+0000 7f14953d5640 10 monclient: init 2023-10-30T20:24:07.813 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f750e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7508037670 0x7f750802f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.814 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f7987fff640 1 --2- 172.21.15.69:0/3531338428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79800c3d00 0x7f79800c4110 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f798002bc00 tx=0x7f797402de20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74210a8df9f57669 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.814 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7fa6377fe640 1 --2- 172.21.15.69:0/3426682588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6381513b0 0x7fa6380aae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f14953d5640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f14953d5640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-10-30T20:24:07.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f14953d5640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f14900b1290 0x7f149006ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f14953d5640 10 monclient(hunting): picked mon.noname-c con 0x7f14900b1290 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:07.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f14953d5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f14900ae140 0x7f149006f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f14953d5640 10 monclient(hunting): picked mon.noname-a con 0x7f14900ae140 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:07.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f14953d5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1490070110 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:07.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f14953d5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1490071170 con 0x7f1490070110 2023-10-30T20:24:07.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f14953d5640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f14953d5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1490074040 con 0x7f14900ae140 2023-10-30T20:24:07.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f14953d5640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f14953d5640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1490071e70 con 0x7f14900b1290 2023-10-30T20:24:07.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f14953d5640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f14953d5640 10 monclient(hunting): authenticate will time out at 1571.008386s 2023-10-30T20:24:07.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f750f59e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f750815b930 0x7f750815bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36562/0 (socket says 172.21.15.69:36562) 2023-10-30T20:24:07.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f750f59e640 1 -- 172.21.15.69:0/3954807367 learned_addr learned my addr 172.21.15.69:0/3954807367 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f750f59e640 10 monclient(hunting): get_auth_request con 0x7f750815b930 auth_method 0 2023-10-30T20:24:07.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f750f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f750f59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.719+0000 7f750f59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.720+0000 7f750e59c640 10 monclient(hunting): get_auth_request con 0x7f7508037670 auth_method 0 2023-10-30T20:24:07.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.720+0000 7f750e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.720+0000 7f750e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.720+0000 7f750e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.720+0000 7fa6377fe640 10 monclient(hunting): get_auth_request con 0x7fa6381513b0 auth_method 0 2023-10-30T20:24:07.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.720+0000 7fa6377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.720+0000 7fa6377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.720+0000 7fa6377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.721+0000 7f750f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.721+0000 7f750f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.721+0000 7f750f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.721+0000 7fa6377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.823 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.721+0000 7fa6377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.823 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.721+0000 7fa6377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.823 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.721+0000 7f750e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.823 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.721+0000 7f750e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.824 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.721+0000 7f750e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.824 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7fa636ffd640 1 --2- 172.21.15.69:0/3426682588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa638151df0 0x7fa63813de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.824 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7fa636ffd640 10 monclient(hunting): get_auth_request con 0x7fa638151df0 auth_method 0 2023-10-30T20:24:07.824 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7fa636ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7fa636ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7fa636ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7f750f59e640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2023-10-30T20:24:07.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7f750f59e640 1 -- 172.21.15.69:0/3954807367 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750802b200 msgr2=0x7f750802f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7f750f59e640 1 --2- 172.21.15.69:0/3954807367 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750802b200 0x7f750802f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7f750f59e640 1 -- 172.21.15.69:0/3954807367 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7508037670 msgr2=0x7f750802f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7f750f59e640 1 --2- 172.21.15.69:0/3954807367 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7508037670 0x7f750802f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7f750f59e640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7f750f59e640 1 monclient: found mon.noname-b 2023-10-30T20:24:07.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7f750f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7f750f59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.724+0000 7f750f59e640 1 -- 172.21.15.69:0/3954807367 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750802fe50 con 0x7f750815b930 2023-10-30T20:24:07.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.725+0000 7fa6377fe640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2023-10-30T20:24:07.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.725+0000 7fa6377fe640 1 -- 172.21.15.69:0/3426682588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa638151df0 msgr2=0x7fa63813de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.725+0000 7fa6377fe640 1 --2- 172.21.15.69:0/3426682588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa638151df0 0x7fa63813de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.725+0000 7fa6377fe640 1 -- 172.21.15.69:0/3426682588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6381521d0 msgr2=0x7fa63813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.725+0000 7fa6377fe640 1 --2- 172.21.15.69:0/3426682588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6381521d0 0x7fa63813d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.725+0000 7fa6377fe640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.725+0000 7fa6377fe640 1 monclient: found mon.noname-a 2023-10-30T20:24:07.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.725+0000 7fa6377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.725+0000 7fa6377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.725+0000 7fa6377fe640 1 -- 172.21.15.69:0/3426682588 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63813e3b0 con 0x7fa6381513b0 2023-10-30T20:24:07.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7fe33f7fe640 1 --2- 172.21.15.69:0/1516023723 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3400a8e40 0x7fe34013efc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe3400b8030 tx=0x7fe33002b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0a8fdddcac562b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.832 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc1ffb640 1 -- 172.21.15.69:0/860255227 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb802e050 con 0x7f6bbc0070a0 2023-10-30T20:24:07.833 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.833 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:07.833 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc1ffb640 10 monclient: dump: 2023-10-30T20:24:07.833 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.833 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.834 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.834 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.834 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.834 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.834 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.835 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.835 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.835 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc1ffb640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc1ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc3fff640 5 monclient: authenticate success, global_id 4305 2023-10-30T20:24:07.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:07.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc1ffb640 1 -- 172.21.15.69:0/860255227 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb8030710 con 0x7f6bbc0070a0 2023-10-30T20:24:07.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc3fff640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:07.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc3fff640 4 set_mon_vals no callback set 2023-10-30T20:24:07.837 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc1ffb640 1 -- 172.21.15.69:0/860255227 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8030ab0 con 0x7f6bbc0070a0 2023-10-30T20:24:07.837 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.837 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:07.837 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.726+0000 7f6bc1ffb640 10 monclient: dump: 2023-10-30T20:24:07.837 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.837 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.838 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.838 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.838 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.838 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.838 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.838 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.839 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.839 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.727+0000 7f6bc3fff640 10 monclient: shutdown 2023-10-30T20:24:07.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.727+0000 7f6bc3fff640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.727+0000 7f6bc3fff640 1 -- 172.21.15.69:0/860255227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bbc0070a0 msgr2=0x7f6bbc0338a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.727+0000 7f6bc3fff640 1 --2- 172.21.15.69:0/860255227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bbc0070a0 0x7f6bbc0338a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc010060 tx=0x7f6bb802b6e0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.727+0000 7f6bc3fff640 1 -- 172.21.15.69:0/860255227 shutdown_connections 2023-10-30T20:24:07.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.727+0000 7f6bc3fff640 1 --2- 172.21.15.69:0/860255227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bbc0307d0 0x7f6bbc035b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.727+0000 7f6bc3fff640 1 --2- 172.21.15.69:0/860255227 >> v2:172.21.15.183:3300/0 conn(0x7f6bbc0360b0 0x7f6bbc038db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.727+0000 7f6bc3fff640 1 --2- 172.21.15.69:0/860255227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bbc0070a0 0x7f6bbc0338a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.727+0000 7f6bc3fff640 1 -- 172.21.15.69:0/860255227 >> 172.21.15.69:0/860255227 conn(0x7f6bbc006040 msgr2=0x7f6bbc006450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.841 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.727+0000 7f6bc3fff640 1 -- 172.21.15.69:0/860255227 shutdown_connections 2023-10-30T20:24:07.841 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.727+0000 7f6bc3fff640 1 -- 172.21.15.69:0/860255227 wait complete. 2023-10-30T20:24:07.841 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.728+0000 7f6bc3fff640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.841 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.728+0000 7f6bc3fff640 10 monclient: monmap: 2023-10-30T20:24:07.841 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.841 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.842 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.842 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.842 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.842 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.842 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.842 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.843 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.843 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.843 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.728+0000 7f6bc3fff640 1 librados: starting msgr at 2023-10-30T20:24:07.843 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.728+0000 7f6bc3fff640 1 librados: starting objecter 2023-10-30T20:24:07.843 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.728+0000 7f6bc3fff640 1 Processor -- start 2023-10-30T20:24:07.844 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.728+0000 7f6bc3fff640 1 -- start start 2023-10-30T20:24:07.844 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.728+0000 7f6bc3fff640 1 librados: setting wanted keys 2023-10-30T20:24:07.844 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.728+0000 7f6bc3fff640 1 librados: calling monclient init 2023-10-30T20:24:07.844 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.728+0000 7f6bc3fff640 10 monclient: init 2023-10-30T20:24:07.844 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.844 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T20:24:07.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bbc0070a0 0x7f6bbc0cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f6bbc0070a0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bbc0360b0 0x7f6bbc0320f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f6bbc0360b0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6bbc0ce8d0 0x7f6bbc0cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f6bbc0ce8d0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0cf8a0 con 0x7f6bbc0360b0 2023-10-30T20:24:07.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0cb660 con 0x7f6bbc0ce8d0 2023-10-30T20:24:07.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0cb940 con 0x7f6bbc0070a0 2023-10-30T20:24:07.847 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.847 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.729+0000 7f6bc3fff640 10 monclient(hunting): authenticate will time out at 1571.018510s 2023-10-30T20:24:07.847 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.731+0000 7f4ec27fc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ec40b1b00 0x7f4ec412f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.848 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.731+0000 7f148effd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f14900b1290 0x7f149006ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.848 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.731+0000 7f148effd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f14900b1290 0x7f149006ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36582/0 (socket says 172.21.15.69:36582) 2023-10-30T20:24:07.848 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.731+0000 7f148effd640 1 -- 172.21.15.69:0/332714988 learned_addr learned my addr 172.21.15.69:0/332714988 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.848 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.731+0000 7f148effd640 10 monclient(hunting): get_auth_request con 0x7f14900b1290 auth_method 0 2023-10-30T20:24:07.849 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.731+0000 7f148effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.849 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.731+0000 7f148effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.849 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.731+0000 7f148effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.849 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.731+0000 7f4ec27fc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ec40b1b00 0x7f4ec412f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36574/0 (socket says 172.21.15.69:36574) 2023-10-30T20:24:07.850 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.731+0000 7f4ec27fc640 1 -- 172.21.15.69:0/3316644440 learned_addr learned my addr 172.21.15.69:0/3316644440 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.850 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.731+0000 7f4ec27fc640 10 monclient(hunting): get_auth_request con 0x7f4ec40b1b00 auth_method 0 2023-10-30T20:24:07.850 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.731+0000 7f4ec27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.850 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.731+0000 7f4ec27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.850 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.731+0000 7f4ec27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.850 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.732+0000 7f4ec27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.851 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.732+0000 7f4ec27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.851 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.732+0000 7f4ec27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.851 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.732+0000 7f148effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.732+0000 7f148effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.732+0000 7f148effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.733+0000 7f4ec37fe640 1 --2- 172.21.15.69:0/3316644440 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec4143220 0x7f4ec40b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.734+0000 7f4ec37fe640 10 monclient(hunting): get_auth_request con 0x7f4ec4143220 auth_method 0 2023-10-30T20:24:07.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.734+0000 7f4ec37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.734+0000 7f4ec37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.734+0000 7f4ec37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.734+0000 7fa6377fe640 1 --2- 172.21.15.69:0/3426682588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6381513b0 0x7fa6380aae50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa62800c5e0 tx=0x7fa62800caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.734+0000 7fa61ffff640 10 client.?.objecter ms_handle_connect 0x7fa6381513b0 2023-10-30T20:24:07.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.734+0000 7fa61ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:07.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.734+0000 7f4ec37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.734+0000 7f4ec37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.734+0000 7f4ec37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.736+0000 7f4ec27fc640 10 monclient(hunting): handle_auth_done global_id 4327 payload 824 2023-10-30T20:24:07.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.736+0000 7f4ec27fc640 1 -- 172.21.15.69:0/3316644440 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec4143220 msgr2=0x7f4ec40b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.736+0000 7f4ec27fc640 1 --2- 172.21.15.69:0/3316644440 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec4143220 0x7f4ec40b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.736+0000 7f4ec27fc640 1 -- 172.21.15.69:0/3316644440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec40a88d0 msgr2=0x7f4ec40ae250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.736+0000 7f4ec27fc640 1 --2- 172.21.15.69:0/3316644440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec40a88d0 0x7f4ec40ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.736+0000 7f4ec27fc640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.857 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.736+0000 7f4ec27fc640 1 monclient: found mon.noname-b 2023-10-30T20:24:07.857 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.736+0000 7f7fb77fe640 1 --2- 172.21.15.69:0/597110672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb813f500 0x7f7fb81425c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009e50 tx=0x7f7fb002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25af0e1a78d24d1d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.736+0000 7f4ec27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.736+0000 7f4ec27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.736+0000 7f4ec27fc640 1 -- 172.21.15.69:0/3316644440 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec412fbe0 con 0x7f4ec40b1b00 2023-10-30T20:24:07.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe33e7fc640 1 -- 172.21.15.69:0/1516023723 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe330031750 con 0x7fe3400a8e40 2023-10-30T20:24:07.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe33e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.860 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe33e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:07.860 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe33e7fc640 10 monclient: dump: 2023-10-30T20:24:07.860 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.860 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.860 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.860 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.861 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.861 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.861 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.861 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.861 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.862 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe33e7fc640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe33e7fc640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe33e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe33e7fc640 1 -- 172.21.15.69:0/1516023723 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3300318c0 con 0x7fe3400a8e40 2023-10-30T20:24:07.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe33e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe33e7fc640 1 -- 172.21.15.69:0/1516023723 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe330031c60 con 0x7fe3400a8e40 2023-10-30T20:24:07.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe345fe9640 5 monclient: authenticate success, global_id 4321 2023-10-30T20:24:07.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe345fe9640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:07.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe345fe9640 4 set_mon_vals no callback set 2023-10-30T20:24:07.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe33e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe33e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.737+0000 7fe33e7fc640 10 monclient: dump: 2023-10-30T20:24:07.864 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.864 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.864 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.864 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.865 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.865 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.865 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.865 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.865 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.865 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.738+0000 7fe345fe9640 10 monclient: shutdown 2023-10-30T20:24:07.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.738+0000 7fe345fe9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.738+0000 7fe345fe9640 1 -- 172.21.15.69:0/1516023723 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3400a8e40 msgr2=0x7fe34013efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.738+0000 7fe345fe9640 1 --2- 172.21.15.69:0/1516023723 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3400a8e40 0x7fe34013efc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe3400b8030 tx=0x7fe33002b7f0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.738+0000 7fe345fe9640 1 -- 172.21.15.69:0/1516023723 shutdown_connections 2023-10-30T20:24:07.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.738+0000 7fe345fe9640 1 --2- 172.21.15.69:0/1516023723 >> v2:172.21.15.69:3300/0 conn(0x7fe34013f500 0x7fe3401425c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.738+0000 7fe345fe9640 1 --2- 172.21.15.69:0/1516023723 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3400a8e40 0x7fe34013efc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.738+0000 7fe345fe9640 1 -- 172.21.15.69:0/1516023723 >> 172.21.15.69:0/1516023723 conn(0x7fe3400b7180 msgr2=0x7fe3400b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.738+0000 7fe345fe9640 1 -- 172.21.15.69:0/1516023723 shutdown_connections 2023-10-30T20:24:07.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.738+0000 7fe345fe9640 1 -- 172.21.15.69:0/1516023723 wait complete. 2023-10-30T20:24:07.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7fe345fe9640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7fe345fe9640 10 monclient: monmap: 2023-10-30T20:24:07.868 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.868 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.868 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.868 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.868 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.868 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.869 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.869 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.869 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.869 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f79867fc640 1 -- 172.21.15.69:0/3531338428 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974032070 con 0x7f79800c3d00 2023-10-30T20:24:07.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f79867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f79867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:07.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f79867fc640 10 monclient: dump: 2023-10-30T20:24:07.871 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.871 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.871 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.871 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.872 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.872 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.872 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.872 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.872 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.873 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f79867fc640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f79867fc640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f79867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f79867fc640 1 -- 172.21.15.69:0/3531338428 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797402ebd0 con 0x7f79800c3d00 2023-10-30T20:24:07.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f798cd67640 5 monclient: authenticate success, global_id 4318 2023-10-30T20:24:07.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f798cd67640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:07.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f79867fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f798cd67640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:07.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f798cd67640 4 set_mon_vals no callback set 2023-10-30T20:24:07.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f79867fc640 1 -- 172.21.15.69:0/3531338428 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974038400 con 0x7f79800c3d00 2023-10-30T20:24:07.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f79867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f79867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f79867fc640 10 monclient: dump: 2023-10-30T20:24:07.875 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.875 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.876 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.876 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.876 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.876 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.876 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.877 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.877 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.877 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7f6bc37fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6bbc0ce8d0 0x7f6bbc0cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7fe345fe9640 1 librados: starting msgr at 2023-10-30T20:24:07.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.739+0000 7fe345fe9640 1 librados: starting objecter 2023-10-30T20:24:07.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7fe345fe9640 1 Processor -- start 2023-10-30T20:24:07.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f6bc2ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bbc0070a0 0x7f6bbc0cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7fe345fe9640 1 -- start start 2023-10-30T20:24:07.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7fe345fe9640 1 librados: setting wanted keys 2023-10-30T20:24:07.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7fe345fe9640 1 librados: calling monclient init 2023-10-30T20:24:07.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7fe345fe9640 10 monclient: init 2023-10-30T20:24:07.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f6bc2ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bbc0070a0 0x7f6bbc0cbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60790/0 (socket says 172.21.15.69:60790) 2023-10-30T20:24:07.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f6bc2ffd640 1 -- 172.21.15.69:0/883138111 learned_addr learned my addr 172.21.15.69:0/883138111 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f6bc2ffd640 10 monclient(hunting): get_auth_request con 0x7f6bbc0070a0 auth_method 0 2023-10-30T20:24:07.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f6bc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f6bc2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f6bc2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f798cd67640 10 monclient: shutdown 2023-10-30T20:24:07.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f798cd67640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f798cd67640 1 -- 172.21.15.69:0/3531338428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79800c3d00 msgr2=0x7f79800c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f798cd67640 1 --2- 172.21.15.69:0/3531338428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79800c3d00 0x7f79800c4110 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f798002bc00 tx=0x7f797402de20 comp rx=0 tx=0).stop 2023-10-30T20:24:07.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f798cd67640 1 -- 172.21.15.69:0/3531338428 shutdown_connections 2023-10-30T20:24:07.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f798cd67640 1 --2- 172.21.15.69:0/3531338428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79800c3d00 0x7f79800c4110 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f798cd67640 1 --2- 172.21.15.69:0/3531338428 >> v2:172.21.15.69:3300/0 conn(0x7f7980034e80 0x7f798002a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f798cd67640 1 -- 172.21.15.69:0/3531338428 >> 172.21.15.69:0/3531338428 conn(0x7f79800060a0 msgr2=0x7f79800064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f798cd67640 1 -- 172.21.15.69:0/3531338428 shutdown_connections 2023-10-30T20:24:07.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7f798cd67640 1 -- 172.21.15.69:0/3531338428 wait complete. 2023-10-30T20:24:07.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7fe345fe9640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7fe345fe9640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T20:24:07.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.740+0000 7fe345fe9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3400a8e40 0x7fe34013c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7fe345fe9640 10 monclient(hunting): picked mon.noname-b con 0x7fe3400a8e40 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7fe345fe9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3400aacb0 0x7fe3401d9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7fe345fe9640 10 monclient(hunting): picked mon.noname-a con 0x7fe3400aacb0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7fe345fe9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3401da4d0 0x7fe340136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7fe345fe9640 10 monclient(hunting): picked mon.noname-c con 0x7fe3401da4d0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7fe345fe9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7fe345fe9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3400aa250 con 0x7fe3400aacb0 2023-10-30T20:24:07.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7fe345fe9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7fe345fe9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3400ab0f0 con 0x7fe3400a8e40 2023-10-30T20:24:07.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7fe345fe9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7fe345fe9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe340141570 con 0x7fe3401da4d0 2023-10-30T20:24:07.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7fe345fe9640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7fe345fe9640 10 monclient(hunting): authenticate will time out at 1571.030026s 2023-10-30T20:24:07.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f798cd67640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f798cd67640 10 monclient: monmap: 2023-10-30T20:24:07.888 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.888 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.888 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.888 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.888 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.889 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.889 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.889 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.889 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.889 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f798cd67640 1 librados: starting msgr at 2023-10-30T20:24:07.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f798cd67640 1 librados: starting objecter 2023-10-30T20:24:07.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f6bc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f6bc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f6bc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f798cd67640 1 Processor -- start 2023-10-30T20:24:07.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f148effd640 10 monclient(hunting): handle_auth_done global_id 4330 payload 274 2023-10-30T20:24:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f148effd640 1 -- 172.21.15.69:0/332714988 >> v2:172.21.15.69:3300/0 conn(0x7f14900ae140 msgr2=0x7f149006f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f148effd640 1 --2- 172.21.15.69:0/332714988 >> v2:172.21.15.69:3300/0 conn(0x7f14900ae140 0x7f149006f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f148effd640 1 -- 172.21.15.69:0/332714988 >> v1:172.21.15.183:6789/0 conn(0x7f1490070110 legacy=0x7f149013b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f148effd640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 1 -- start start 2023-10-30T20:24:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 1 librados: setting wanted keys 2023-10-30T20:24:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 1 librados: calling monclient init 2023-10-30T20:24:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 10 monclient: init 2023-10-30T20:24:07.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f148effd640 1 monclient: found mon.noname-c 2023-10-30T20:24:07.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f148effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f148effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.741+0000 7f148effd640 1 -- 172.21.15.69:0/332714988 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1490071fc0 con 0x7f14900b1290 2023-10-30T20:24:07.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f6bc37fe640 10 monclient(hunting): get_auth_request con 0x7f6bbc0ce8d0 auth_method 0 2023-10-30T20:24:07.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f6bc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f6bc37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f6bc37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T20:24:07.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7980034aa0 0x7f7980031870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 10 monclient(hunting): picked mon.noname-a con 0x7f7980034aa0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79800c3d00 0x7f79800c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 10 monclient(hunting): picked mon.noname-c con 0x7f79800c3d00 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79800bb260 0x7f79800bb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 10 monclient(hunting): picked mon.noname-b con 0x7f79800bb260 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7980000d20 con 0x7f7980034aa0 2023-10-30T20:24:07.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79800c7830 con 0x7f79800bb260 2023-10-30T20:24:07.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79800a0d50 con 0x7f79800c3d00 2023-10-30T20:24:07.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.742+0000 7f798cd67640 10 monclient(hunting): authenticate will time out at 1571.031538s 2023-10-30T20:24:07.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.744+0000 7f6bc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.744+0000 7f6bc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.744+0000 7f6bc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f6bc2ffd640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2023-10-30T20:24:07.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f6bc2ffd640 1 -- 172.21.15.69:0/883138111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6bbc0ce8d0 msgr2=0x7f6bbc0cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f6bc2ffd640 1 --2- 172.21.15.69:0/883138111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6bbc0ce8d0 0x7f6bbc0cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f6bc2ffd640 1 -- 172.21.15.69:0/883138111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bbc0360b0 msgr2=0x7f6bbc0320f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f6bc2ffd640 1 --2- 172.21.15.69:0/883138111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bbc0360b0 0x7f6bbc0320f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f6bc2ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f6bc2ffd640 1 monclient: found mon.noname-c 2023-10-30T20:24:07.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f6bc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f6bc2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f6bc2ffd640 1 -- 172.21.15.69:0/883138111 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc032ca0 con 0x7f6bbc0070a0 2023-10-30T20:24:07.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fb5ffb640 1 -- 172.21.15.69:0/597110672 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb002c9f0 con 0x7f7fb813f500 2023-10-30T20:24:07.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fb5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:07.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fb5ffb640 10 monclient: dump: 2023-10-30T20:24:07.903 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.903 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.904 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.904 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.904 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.904 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.904 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.905 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.905 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.905 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fb5ffb640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fb5ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fb5ffb640 1 -- 172.21.15.69:0/597110672 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb002cb60 con 0x7f7fb813f500 2023-10-30T20:24:07.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fb5ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fb5ffb640 1 -- 172.21.15.69:0/597110672 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb00343f0 con 0x7f7fb813f500 2023-10-30T20:24:07.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fbd390640 5 monclient: authenticate success, global_id 4308 2023-10-30T20:24:07.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fbd390640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:07.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fbd390640 4 set_mon_vals no callback set 2023-10-30T20:24:07.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fb5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:07.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f7fb5ffb640 10 monclient: dump: 2023-10-30T20:24:07.907 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.907 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.908 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.908 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.908 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.908 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.908 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.908 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.909 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.909 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f750f59e640 1 --2- 172.21.15.69:0/3954807367 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f750815b930 0x7f750815bdb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7510066dc0 tx=0x7f7510067340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f74f77fe640 10 client.?.objecter ms_handle_connect 0x7f750815b930 2023-10-30T20:24:07.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.745+0000 7f74f77fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:07.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.746+0000 7f7fbd390640 10 monclient: shutdown 2023-10-30T20:24:07.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.746+0000 7f7fbd390640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.746+0000 7f7fbd390640 1 -- 172.21.15.69:0/597110672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb813f500 msgr2=0x7f7fb81425c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.746+0000 7f7fbd390640 1 --2- 172.21.15.69:0/597110672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb813f500 0x7f7fb81425c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009e50 tx=0x7f7fb002be50 comp rx=0 tx=0).stop 2023-10-30T20:24:07.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.746+0000 7f7fbd390640 1 -- 172.21.15.69:0/597110672 shutdown_connections 2023-10-30T20:24:07.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.746+0000 7f7fbd390640 1 --2- 172.21.15.69:0/597110672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb813f500 0x7f7fb81425c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.746+0000 7f7fbd390640 1 --2- 172.21.15.69:0/597110672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb80a8e40 0x7f7fb813efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.746+0000 7f7fbd390640 1 -- 172.21.15.69:0/597110672 >> 172.21.15.69:0/597110672 conn(0x7f7fb80b7180 msgr2=0x7f7fb80b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.746+0000 7f7fbd390640 1 -- 172.21.15.69:0/597110672 shutdown_connections 2023-10-30T20:24:07.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.746+0000 7f148effd640 1 --2- 172.21.15.69:0/332714988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14900b1290 0x7f149006ec40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f149006e990 tx=0x7f148002e030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0c217f7aa61faaa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.746+0000 7f7fbd390640 1 -- 172.21.15.69:0/597110672 wait complete. 2023-10-30T20:24:07.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.747+0000 7f7fbd390640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.747+0000 7f7fbd390640 10 monclient: monmap: 2023-10-30T20:24:07.913 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.914 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.914 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.914 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.914 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.914 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.914 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.915 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.915 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.915 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.747+0000 7f7fbd390640 1 librados: starting msgr at 2023-10-30T20:24:07.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.747+0000 7f7fbd390640 1 librados: starting objecter 2023-10-30T20:24:07.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7986ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79800c3d00 0x7f79800c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 1 Processor -- start 2023-10-30T20:24:07.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 1 -- start start 2023-10-30T20:24:07.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 1 librados: setting wanted keys 2023-10-30T20:24:07.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 1 librados: calling monclient init 2023-10-30T20:24:07.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 10 monclient: init 2023-10-30T20:24:07.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f6bc2ffd640 1 --2- 172.21.15.69:0/883138111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bbc0070a0 0x7f6bbc0cbd60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8000c00 tx=0x7f6bb80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f6bab7fe640 10 client.?.objecter ms_handle_connect 0x7f6bbc0070a0 2023-10-30T20:24:07.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f6bab7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:07.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T20:24:07.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb80a8e40 0x7f7fb813ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 10 monclient(hunting): picked mon.noname-a con 0x7f7fb80a8e40 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb813faa0 0x7f7fb81404b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 10 monclient(hunting): picked mon.noname-c con 0x7f7fb813faa0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fb813b4d0 0x7f7fb813b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 10 monclient(hunting): picked mon.noname-b con 0x7f7fb813b4d0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.748+0000 7f7fbd390640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb80ae010 con 0x7f7fb80a8e40 2023-10-30T20:24:07.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.749+0000 7f7fbd390640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.749+0000 7f7fbd390640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb81410f0 con 0x7f7fb813b4d0 2023-10-30T20:24:07.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.749+0000 7f7fbd390640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.749+0000 7f7fbd390640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb8140bd0 con 0x7f7fb813faa0 2023-10-30T20:24:07.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.749+0000 7f7fbd390640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.749+0000 7f7fbd390640 10 monclient(hunting): authenticate will time out at 1571.037853s 2023-10-30T20:24:07.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.750+0000 7f79877fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7980034aa0 0x7f7980031870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.751+0000 7f7986ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79800c3d00 0x7f79800c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60818/0 (socket says 172.21.15.69:60818) 2023-10-30T20:24:07.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.751+0000 7f7986ffd640 1 -- 172.21.15.69:0/276416905 learned_addr learned my addr 172.21.15.69:0/276416905 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.751+0000 7f7986ffd640 10 monclient(hunting): get_auth_request con 0x7f79800c3d00 auth_method 0 2023-10-30T20:24:07.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.751+0000 7f7986ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.751+0000 7f7986ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.751+0000 7f7986ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.751+0000 7fe33ffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3401da4d0 0x7fe340136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.752+0000 7f7986ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.752+0000 7f7986ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.752+0000 7f7986ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.752+0000 7fe33ffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3401da4d0 0x7fe340136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60806/0 (socket says 172.21.15.69:60806) 2023-10-30T20:24:07.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.752+0000 7fe33ffff640 1 -- 172.21.15.69:0/298708272 learned_addr learned my addr 172.21.15.69:0/298708272 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.752+0000 7fe33ffff640 10 monclient(hunting): get_auth_request con 0x7fe3401da4d0 auth_method 0 2023-10-30T20:24:07.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.752+0000 7fe33ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.752+0000 7fe33ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.752+0000 7fe33ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.753+0000 7f79877fe640 10 monclient(hunting): get_auth_request con 0x7f7980034aa0 auth_method 0 2023-10-30T20:24:07.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.753+0000 7f79877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.753+0000 7f79877fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.753+0000 7f79877fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.753+0000 7f79877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.753+0000 7f79877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7f79877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7f4ec27fc640 1 --2- 172.21.15.69:0/3316644440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ec40b1b00 0x7f4ec412f6a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc036040 tx=0x7f4ebc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7f4eabfff640 10 client.?.objecter ms_handle_connect 0x7f4ec40b1b00 2023-10-30T20:24:07.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7f4eabfff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:07.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa61ffff640 1 -- 172.21.15.69:0/3426682588 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628004220 con 0x7fa6381513b0 2023-10-30T20:24:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa61ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa61ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa61ffff640 10 monclient: dump: 2023-10-30T20:24:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.932 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa61ffff640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa61ffff640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa61ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa61ffff640 1 -- 172.21.15.69:0/3426682588 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62800d040 con 0x7fa6381513b0 2023-10-30T20:24:07.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa61ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa61ffff640 1 -- 172.21.15.69:0/3426682588 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628003610 con 0x7fa6381513b0 2023-10-30T20:24:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa63dba9640 5 monclient: authenticate success, global_id 4311 2023-10-30T20:24:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa63dba9640 10 monclient: _renew_subs 2023-10-30T20:24:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa63dba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa63dba9640 1 -- 172.21.15.69:0/3426682588 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63813d630 con 0x7fa6381513b0 2023-10-30T20:24:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa61ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa61ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:07.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa61ffff640 10 monclient: dump: 2023-10-30T20:24:07.934 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.934 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.934 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.936 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa63dba9640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa63dba9640 10 monclient: _renew_subs 2023-10-30T20:24:07.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa63dba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa63dba9640 1 -- 172.21.15.69:0/3426682588 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6381e3d50 con 0x7fa6381513b0 2023-10-30T20:24:07.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa635ffb640 4 set_mon_vals no callback set 2023-10-30T20:24:07.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fa63dba9640 1 librados: init done 2023-10-30T20:24:07.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.754+0000 7fe33effd640 1 --2- 172.21.15.69:0/298708272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3400aacb0 0x7fe3401d9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.755+0000 7fa63dba9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:07.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.755+0000 7fa63dba9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:07.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.755+0000 7fa63dba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:07.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.755+0000 7fa63dba9640 1 -- 172.21.15.69:0/3426682588 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa63804e9b0 con 0x7fa6381513b0 2023-10-30T20:24:07.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.755+0000 7f7986ffd640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2023-10-30T20:24:07.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.755+0000 7f7986ffd640 1 -- 172.21.15.69:0/276416905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79800bb260 msgr2=0x7f79800bb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.755+0000 7f7986ffd640 1 --2- 172.21.15.69:0/276416905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79800bb260 0x7f79800bb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.755+0000 7f7986ffd640 1 -- 172.21.15.69:0/276416905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7980034aa0 msgr2=0x7f7980031870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.755+0000 7f7986ffd640 1 --2- 172.21.15.69:0/276416905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7980034aa0 0x7f7980031870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.755+0000 7f7986ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.755+0000 7f7986ffd640 1 monclient: found mon.noname-c 2023-10-30T20:24:07.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.755+0000 7f7986ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.755+0000 7f7986ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.755+0000 7f7986ffd640 1 -- 172.21.15.69:0/276416905 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79800a1030 con 0x7f79800c3d00 2023-10-30T20:24:07.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.756+0000 7fe33f7fe640 1 --2- 172.21.15.69:0/298708272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3400a8e40 0x7fe34013c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.758+0000 7fe33effd640 10 monclient(hunting): get_auth_request con 0x7fe3400aacb0 auth_method 0 2023-10-30T20:24:07.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.758+0000 7fe33effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.758+0000 7fe33effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.758+0000 7fe33effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.758+0000 7fe33ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.758+0000 7fe33ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.758+0000 7fe33ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.758+0000 7f7986ffd640 1 --2- 172.21.15.69:0/276416905 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79800c3d00 0x7f79800c72f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7988066dc0 tx=0x7f7988067340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.758+0000 7f7973fff640 10 client.?.objecter ms_handle_connect 0x7f79800c3d00 2023-10-30T20:24:07.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.758+0000 7f7973fff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:07.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.759+0000 7fe33f7fe640 10 monclient(hunting): get_auth_request con 0x7fe3400a8e40 auth_method 0 2023-10-30T20:24:07.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.759+0000 7fe33f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.759+0000 7fe33f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.759+0000 7fe33f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.759+0000 7fe33effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.759+0000 7fe33effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.759+0000 7fe33effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.759+0000 7fe33f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.759+0000 7fe33f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.759+0000 7fe33f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.761+0000 7fe33ffff640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2023-10-30T20:24:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.761+0000 7fe33ffff640 1 -- 172.21.15.69:0/298708272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3400a8e40 msgr2=0x7fe34013c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.761+0000 7fe33ffff640 1 --2- 172.21.15.69:0/298708272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3400a8e40 0x7fe34013c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.761+0000 7fe33ffff640 1 -- 172.21.15.69:0/298708272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3400aacb0 msgr2=0x7fe3401d9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.761+0000 7fe33ffff640 1 --2- 172.21.15.69:0/298708272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3400aacb0 0x7fe3401d9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.761+0000 7fe33ffff640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.761+0000 7fe33ffff640 1 monclient: found mon.noname-c 2023-10-30T20:24:07.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.761+0000 7fe33ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.761+0000 7fe33ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.761+0000 7fe33ffff640 1 -- 172.21.15.69:0/298708272 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3400ab7c0 con 0x7fe3401da4d0 2023-10-30T20:24:07.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f7fb77fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fb813b4d0 0x7f7fb813b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f74f77fe640 1 -- 172.21.15.69:0/3954807367 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75100723d0 con 0x7f750815b930 2023-10-30T20:24:07.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f74f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f74f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:07.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f74f77fe640 10 monclient: dump: 2023-10-30T20:24:07.949 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.950 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.950 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.950 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.950 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.950 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.950 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.951 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.951 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.951 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f74f77fe640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f74f77fe640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f74f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f75149ed640 5 monclient: authenticate success, global_id 4324 2023-10-30T20:24:07.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f74f77fe640 1 -- 172.21.15.69:0/3954807367 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7510072040 con 0x7f750815b930 2023-10-30T20:24:07.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f74f77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f74f77fe640 1 -- 172.21.15.69:0/3954807367 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510092400 con 0x7f750815b930 2023-10-30T20:24:07.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f74f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f74f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f74f77fe640 10 monclient: dump: 2023-10-30T20:24:07.953 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.953 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.953 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.953 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.954 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.954 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.954 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.954 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.954 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.955 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.763+0000 7f750d59a640 4 set_mon_vals no callback set 2023-10-30T20:24:07.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.764+0000 7f75149ed640 10 monclient: _renew_subs 2023-10-30T20:24:07.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.764+0000 7f75149ed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.764+0000 7f75149ed640 1 -- 172.21.15.69:0/3954807367 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7508030050 con 0x7f750815b930 2023-10-30T20:24:07.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.764+0000 7f75149ed640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.764+0000 7f75149ed640 10 monclient: _renew_subs 2023-10-30T20:24:07.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.764+0000 7f75149ed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.764+0000 7f75149ed640 1 -- 172.21.15.69:0/3954807367 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750815c4d0 con 0x7f750815b930 2023-10-30T20:24:07.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.764+0000 7f7fb77fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fb813b4d0 0x7f7fb813b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:36618/0 (socket says 172.21.15.69:36618) 2023-10-30T20:24:07.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.764+0000 7f7fb77fe640 1 -- 172.21.15.69:0/314820218 learned_addr learned my addr 172.21.15.69:0/314820218 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:07.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.764+0000 7f7fb77fe640 10 monclient(hunting): get_auth_request con 0x7f7fb813b4d0 auth_method 0 2023-10-30T20:24:07.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.764+0000 7f7fb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.764+0000 7f7fb77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.764+0000 7f75149ed640 1 librados: init done 2023-10-30T20:24:07.958 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.764+0000 7f7fb77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.958 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.765+0000 7fe33ffff640 1 --2- 172.21.15.69:0/298708272 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3401da4d0 0x7fe340136f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe33800ba10 tx=0x7fe33800bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:07.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.765+0000 7fe327fff640 10 client.?.objecter ms_handle_connect 0x7fe3401da4d0 2023-10-30T20:24:07.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.765+0000 7fe327fff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:07.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.765+0000 7f75149ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:07.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.765+0000 7f75149ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:07.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.765+0000 7f75149ed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.765+0000 7f75149ed640 1 -- 172.21.15.69:0/3954807367 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75080057f0 con 0x7f750815b930 2023-10-30T20:24:07.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f7fb6ffd640 1 --2- 172.21.15.69:0/314820218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb80a8e40 0x7f7fb813ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bab7fe640 1 -- 172.21.15.69:0/883138111 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8032bc0 con 0x7f6bbc0070a0 2023-10-30T20:24:07.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:07.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bab7fe640 10 monclient: dump: 2023-10-30T20:24:07.962 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.962 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.962 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.962 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.962 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.963 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.963 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.963 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.963 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.963 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bab7fe640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bab7fe640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bab7fe640 1 -- 172.21.15.69:0/883138111 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb8032d50 con 0x7f6bbc0070a0 2023-10-30T20:24:07.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bab7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bab7fe640 1 -- 172.21.15.69:0/883138111 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb803c6b0 con 0x7f6bbc0070a0 2023-10-30T20:24:07.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:07.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bab7fe640 10 monclient: dump: 2023-10-30T20:24:07.965 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.965 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.965 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.965 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.966 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.966 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.966 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.966 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.966 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.966 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bc3fff640 5 monclient: authenticate success, global_id 4289 2023-10-30T20:24:07.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bc3fff640 10 monclient: _renew_subs 2023-10-30T20:24:07.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bc17fa640 4 set_mon_vals no callback set 2023-10-30T20:24:07.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bc3fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bc3fff640 1 -- 172.21.15.69:0/883138111 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bbc0ccd40 con 0x7f6bbc0070a0 2023-10-30T20:24:07.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bc3fff640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:07.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bc3fff640 10 monclient: _renew_subs 2023-10-30T20:24:07.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bc3fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bc3fff640 1 -- 172.21.15.69:0/883138111 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bbc0b7a30 con 0x7f6bbc0070a0 2023-10-30T20:24:07.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.767+0000 7f6bc3fff640 1 librados: init done 2023-10-30T20:24:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.768+0000 7f7fb67fc640 1 --2- 172.21.15.69:0/314820218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb813faa0 0x7f7fb81404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.768+0000 7f7fb6ffd640 10 monclient(hunting): get_auth_request con 0x7f7fb80a8e40 auth_method 0 2023-10-30T20:24:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.768+0000 7f7fb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.768+0000 7f7fb6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.768+0000 7f7fb6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.769+0000 7f6bc3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:07.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.769+0000 7f6bc3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:07.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.769+0000 7f6bc3fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:07.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.769+0000 7f6bc3fff640 1 -- 172.21.15.69:0/883138111 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bbc0ccf10 con 0x7f6bbc0070a0 2023-10-30T20:24:07.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.769+0000 7f7fb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:07.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.769+0000 7f7fb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:07.972 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.769+0000 7f7fb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:07.973 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f148e7fc640 1 -- 172.21.15.69:0/332714988 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148002ede0 con 0x7f14900b1290 2023-10-30T20:24:07.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f148e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f148e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:07.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f148e7fc640 10 monclient: dump: 2023-10-30T20:24:07.974 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.974 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.975 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.975 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.975 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.975 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.975 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.975 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.976 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.976 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.976 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f148e7fc640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.976 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f148e7fc640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.976 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f148e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.976 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f14953d5640 5 monclient: authenticate success, global_id 4330 2023-10-30T20:24:07.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f148e7fc640 1 -- 172.21.15.69:0/332714988 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1480004030 con 0x7f14900b1290 2023-10-30T20:24:07.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f14953d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:07.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f148e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f148e7fc640 1 -- 172.21.15.69:0/332714988 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148002ede0 con 0x7f14900b1290 2023-10-30T20:24:07.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f148e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f148e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f148e7fc640 10 monclient: dump: 2023-10-30T20:24:07.978 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.978 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.978 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.978 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.979 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.979 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.979 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.979 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.980 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.980 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f14953d5640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:07.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.770+0000 7f14953d5640 4 set_mon_vals no callback set 2023-10-30T20:24:07.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.771+0000 7f14953d5640 10 monclient: shutdown 2023-10-30T20:24:07.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.771+0000 7f14953d5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:07.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.771+0000 7f14953d5640 1 -- 172.21.15.69:0/332714988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14900b1290 msgr2=0x7f149006ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.771+0000 7f14953d5640 1 --2- 172.21.15.69:0/332714988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14900b1290 0x7f149006ec40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f149006e990 tx=0x7f148002e030 comp rx=0 tx=0).stop 2023-10-30T20:24:07.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.771+0000 7f14953d5640 1 -- 172.21.15.69:0/332714988 shutdown_connections 2023-10-30T20:24:07.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.771+0000 7f14953d5640 1 --2- 172.21.15.69:0/332714988 >> v2:172.21.15.69:3300/0 conn(0x7f14900ae140 0x7f149006f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.771+0000 7f14953d5640 1 --2- 172.21.15.69:0/332714988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14900b1290 0x7f149006ec40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.771+0000 7f14953d5640 1 -- 172.21.15.69:0/332714988 >> 172.21.15.69:0/332714988 conn(0x7f14900b4460 msgr2=0x7f14900b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:07.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.771+0000 7f14953d5640 1 -- 172.21.15.69:0/332714988 shutdown_connections 2023-10-30T20:24:07.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.771+0000 7f14953d5640 1 -- 172.21.15.69:0/332714988 wait complete. 2023-10-30T20:24:07.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.772+0000 7f7fb67fc640 10 monclient(hunting): get_auth_request con 0x7f7fb813faa0 auth_method 0 2023-10-30T20:24:07.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.772+0000 7f7fb67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:07.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.772+0000 7f7fb67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:07.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.772+0000 7f7fb67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:07.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.772+0000 7f14953d5640 10 monclient: build_initial_monmap 2023-10-30T20:24:07.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.772+0000 7f14953d5640 10 monclient: monmap: 2023-10-30T20:24:07.984 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:07.984 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:07.984 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:07.985 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:07.985 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:07.985 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:07.985 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:07.985 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:07.986 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:07.986 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f14953d5640 1 librados: starting msgr at 2023-10-30T20:24:07.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f14953d5640 1 librados: starting objecter 2023-10-30T20:24:07.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f14953d5640 1 Processor -- start 2023-10-30T20:24:07.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f14953d5640 1 -- start start 2023-10-30T20:24:07.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f14953d5640 1 librados: setting wanted keys 2023-10-30T20:24:07.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f14953d5640 1 librados: calling monclient init 2023-10-30T20:24:07.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f14953d5640 10 monclient: init 2023-10-30T20:24:07.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f7fb77fe640 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2023-10-30T20:24:07.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f7fb77fe640 1 -- 172.21.15.69:0/314820218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb813faa0 msgr2=0x7f7fb81404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f7fb77fe640 1 --2- 172.21.15.69:0/314820218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb813faa0 0x7f7fb81404b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f7fb77fe640 1 -- 172.21.15.69:0/314820218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb80a8e40 msgr2=0x7f7fb813ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:07.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f7fb77fe640 1 --2- 172.21.15.69:0/314820218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb80a8e40 0x7f7fb813ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:07.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f7fb77fe640 10 monclient: _finish_hunting 0 2023-10-30T20:24:07.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f7fb77fe640 1 monclient: found mon.noname-b 2023-10-30T20:24:07.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f7fb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:07.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f7fb77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:24:07.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.773+0000 7f7fb77fe640 1 -- 172.21.15.69:0/314820218 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb81d6bf0 con 0x7f7fb813b4d0 2023-10-30T20:24:07.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:07.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:24:07.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1490070110 0x7f14900a9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 10 monclient(hunting): picked mon.noname-b con 0x7f1490070110 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:07.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14900ae140 0x7f1490134200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 10 monclient(hunting): picked mon.noname-c con 0x7f14900ae140 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:07.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14900b1290 0x7f1490134740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:07.993 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 10 monclient(hunting): picked mon.noname-a con 0x7f14900b1290 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:07.993 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.993 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149006e260 con 0x7f14900b1290 2023-10-30T20:24:07.993 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.993 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14900aaac0 con 0x7f1490070110 2023-10-30T20:24:07.993 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:07.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14900ab930 con 0x7f14900ae140 2023-10-30T20:24:07.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:07.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.774+0000 7f14953d5640 10 monclient(hunting): authenticate will time out at 1571.063499s 2023-10-30T20:24:07.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4eabfff640 1 -- 172.21.15.69:0/3316644440 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc03e450 con 0x7f4ec40b1b00 2023-10-30T20:24:07.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4eabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4eabfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:07.996 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4eabfff640 10 monclient: dump: 2023-10-30T20:24:07.996 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:07.996 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:07.996 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.996 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:07.997 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:07.997 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:07.997 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:07.997 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:07.997 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:07.997 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:07.998 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4eabfff640 10 monclient: _finish_auth 0 2023-10-30T20:24:07.998 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4eabfff640 10 monclient: _check_auth_tickets 2023-10-30T20:24:07.998 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4eabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:07.998 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4eabfff640 1 -- 172.21.15.69:0/3316644440 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc03e5c0 con 0x7f4ec40b1b00 2023-10-30T20:24:07.998 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4eabfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:07.999 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4eabfff640 1 -- 172.21.15.69:0/3316644440 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc03e980 con 0x7f4ec40b1b00 2023-10-30T20:24:07.999 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4eabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:07.999 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4eabfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:07.999 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4eabfff640 10 monclient: dump: 2023-10-30T20:24:07.999 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:08.000 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:08.000 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.000 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.000 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:08.000 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:08.001 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:08.001 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:08.001 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:08.001 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:08.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4ec99a9640 5 monclient: authenticate success, global_id 4327 2023-10-30T20:24:08.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4ec99a9640 10 monclient: _renew_subs 2023-10-30T20:24:08.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4ec99a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:08.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4ec99a9640 1 -- 172.21.15.69:0/3316644440 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec40b11f0 con 0x7f4ec40b1b00 2023-10-30T20:24:08.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.776+0000 7f4ec17fa640 4 set_mon_vals no callback set 2023-10-30T20:24:08.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.777+0000 7f4ec99a9640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:08.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.777+0000 7f4ec99a9640 10 monclient: _renew_subs 2023-10-30T20:24:08.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.777+0000 7f4ec99a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:08.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.777+0000 7f4ec99a9640 1 -- 172.21.15.69:0/3316644440 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec413ff20 con 0x7f4ec40b1b00 2023-10-30T20:24:08.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.777+0000 7f4ec99a9640 1 librados: init done 2023-10-30T20:24:08.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.777+0000 7f4ec99a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.777+0000 7f4ec99a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.004 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.777+0000 7f4ec99a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:08.004 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.777+0000 7f4ec99a9640 1 -- 172.21.15.69:0/3316644440 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec40b13f0 con 0x7f4ec40b1b00 2023-10-30T20:24:08.005 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f7973fff640 1 -- 172.21.15.69:0/276416905 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79880723d0 con 0x7f79800c3d00 2023-10-30T20:24:08.005 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f7973fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:08.005 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f7973fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:08.005 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f7973fff640 10 monclient: dump: 2023-10-30T20:24:08.006 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:08.006 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:08.006 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.006 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.006 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:08.007 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:08.007 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:08.007 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:08.007 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:08.007 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:08.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f7973fff640 10 monclient: _finish_auth 0 2023-10-30T20:24:08.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f7973fff640 10 monclient: _check_auth_tickets 2023-10-30T20:24:08.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f7973fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:08.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f7973fff640 1 -- 172.21.15.69:0/276416905 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7988072040 con 0x7f79800c3d00 2023-10-30T20:24:08.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f7973fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:08.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f7973fff640 1 -- 172.21.15.69:0/276416905 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7988092400 con 0x7f79800c3d00 2023-10-30T20:24:08.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f7973fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:08.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f7973fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:08.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f7973fff640 10 monclient: dump: 2023-10-30T20:24:08.009 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:08.009 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:08.010 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.010 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.010 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:08.010 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:08.010 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:08.011 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:08.011 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:08.011 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:08.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f7985ffb640 4 set_mon_vals no callback set 2023-10-30T20:24:08.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f798cd67640 5 monclient: authenticate success, global_id 4292 2023-10-30T20:24:08.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f798cd67640 10 monclient: _renew_subs 2023-10-30T20:24:08.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f798cd67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:08.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f798cd67640 1 -- 172.21.15.69:0/276416905 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79800c7a80 con 0x7f79800c3d00 2023-10-30T20:24:08.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f798cd67640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:08.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f798cd67640 10 monclient: _renew_subs 2023-10-30T20:24:08.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f798cd67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:08.013 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f798cd67640 1 -- 172.21.15.69:0/276416905 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7980160130 con 0x7f79800c3d00 2023-10-30T20:24:08.013 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.779+0000 7f798cd67640 1 librados: init done 2023-10-30T20:24:08.013 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.780+0000 7f798cd67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.013 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.780+0000 7f798cd67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.013 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.780+0000 7f798cd67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:08.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:07.780+0000 7f798cd67640 1 -- 172.21.15.69:0/276416905 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79800a1230 con 0x7f79800c3d00 2023-10-30T20:24:08.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe327fff640 1 -- 172.21.15.69:0/298708272 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33800cb40 con 0x7fe3401da4d0 2023-10-30T20:24:08.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe327fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:08.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe327fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:08.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe327fff640 10 monclient: dump: 2023-10-30T20:24:08.015 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:08.016 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:08.016 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.016 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.016 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:08.016 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:08.016 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:08.017 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:08.017 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:08.017 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:08.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe327fff640 10 monclient: _finish_auth 0 2023-10-30T20:24:08.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe327fff640 10 monclient: _check_auth_tickets 2023-10-30T20:24:08.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe327fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:08.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe327fff640 1 -- 172.21.15.69:0/298708272 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33800ccb0 con 0x7fe3401da4d0 2023-10-30T20:24:08.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe345fe9640 5 monclient: authenticate success, global_id 4295 2023-10-30T20:24:08.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe345fe9640 10 monclient: _renew_subs 2023-10-30T20:24:08.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe345fe9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:08.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe345fe9640 1 -- 172.21.15.69:0/298708272 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe340141170 con 0x7fe3401da4d0 2023-10-30T20:24:08.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe327fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:08.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe327fff640 1 -- 172.21.15.69:0/298708272 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338014680 con 0x7fe3401da4d0 2023-10-30T20:24:08.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe327fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:08.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe327fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:08.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe327fff640 10 monclient: dump: 2023-10-30T20:24:08.020 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:08.020 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:08.020 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.020 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.020 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:08.021 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:08.021 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:08.021 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:08.021 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:08.021 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:08.021 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe345fe9640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:08.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe345fe9640 10 monclient: _renew_subs 2023-10-30T20:24:08.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe345fe9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:08.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe345fe9640 1 -- 172.21.15.69:0/298708272 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3401dab30 con 0x7fe3401da4d0 2023-10-30T20:24:08.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe345fe9640 1 librados: init done 2023-10-30T20:24:08.023 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.008+0000 7fe33dffb640 4 set_mon_vals no callback set 2023-10-30T20:24:08.023 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.009+0000 7fe345fe9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.023 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.009+0000 7fe345fe9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.023 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.009+0000 7fe345fe9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:08.023 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.009+0000 7fe345fe9640 1 -- 172.21.15.69:0/298708272 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe340141720 con 0x7fe3401da4d0 2023-10-30T20:24:08.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.020+0000 7f74f77fe640 1 -- 172.21.15.69:0/3954807367 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f751009c3f0 con 0x7f750815b930 2023-10-30T20:24:08.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.020+0000 7f74f77fe640 1 --2- 172.21.15.69:0/3954807367 >> v2:172.21.15.69:6800/1876307353 conn(0x7f74fc06c810 0x7f74fc06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:08.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.020+0000 7f750ed9d640 1 --2- 172.21.15.69:0/3954807367 >> v2:172.21.15.69:6800/1876307353 conn(0x7f74fc06c810 0x7f74fc06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:08.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.020+0000 7f750ed9d640 10 monclient: get_auth_request con 0x7f74fc06c810 auth_method 0 2023-10-30T20:24:08.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.021+0000 7f750ed9d640 1 --2- 172.21.15.69:0/3954807367 >> v2:172.21.15.69:6800/1876307353 conn(0x7f74fc06c810 0x7f74fc06ecd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7500009670 tx=0x7f7500006bf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:08.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.021+0000 7f74f77fe640 10 client.4324.objecter ms_handle_connect 0x7f74fc06c810 2023-10-30T20:24:08.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.021+0000 7f148f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14900b1290 0x7f1490134740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:08.026 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.021+0000 7f6bab7fe640 1 -- 172.21.15.69:0/883138111 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f6bb8039040 con 0x7f6bbc0070a0 2023-10-30T20:24:08.026 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.022+0000 7f6bab7fe640 1 --2- 172.21.15.69:0/883138111 >> v2:172.21.15.69:6800/1876307353 conn(0x7f6bb006c7c0 0x7f6bb006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:08.026 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.022+0000 7fa61ffff640 1 -- 172.21.15.69:0/3426682588 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fa628003780 con 0x7fa6381513b0 2023-10-30T20:24:08.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.022+0000 7f6bc27fc640 1 --2- 172.21.15.69:0/883138111 >> v2:172.21.15.69:6800/1876307353 conn(0x7f6bb006c7c0 0x7f6bb006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:08.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.022+0000 7f6bc27fc640 10 monclient: get_auth_request con 0x7f6bb006c7c0 auth_method 0 2023-10-30T20:24:08.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.022+0000 7fa61ffff640 1 --2- 172.21.15.69:0/3426682588 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa624075020 0x7fa6240774e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:08.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.022+0000 7fa636ffd640 1 --2- 172.21.15.69:0/3426682588 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa624075020 0x7fa6240774e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:08.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.022+0000 7fa636ffd640 10 monclient: get_auth_request con 0x7fa624075020 auth_method 0 2023-10-30T20:24:08.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.022+0000 7f6bab7fe640 10 client.4289.objecter ms_handle_connect 0x7f6bb006c7c0 2023-10-30T20:24:08.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.022+0000 7f6bc27fc640 1 --2- 172.21.15.69:0/883138111 >> v2:172.21.15.69:6800/1876307353 conn(0x7f6bb006c7c0 0x7f6bb006ec80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc0328a0 tx=0x7f6bb4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:08.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.022+0000 7fa636ffd640 1 --2- 172.21.15.69:0/3426682588 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa624075020 0x7fa6240774e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa62c034000 tx=0x7fa62c02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:08.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.022+0000 7fa61ffff640 10 client.4311.objecter ms_handle_connect 0x7fa624075020 2023-10-30T20:24:08.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.039+0000 7f6bab7fe640 1 -- 172.21.15.69:0/883138111 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f6bb8079e60 con 0x7f6bbc0070a0 2023-10-30T20:24:08.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.039+0000 7f6bab7fe640 10 client.4289.objecter ms_dispatch 0x7f6bc4000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:08.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.039+0000 7f6bab7fe640 3 client.4289.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:08.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.039+0000 7f6bab7fe640 3 client.4289.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:08.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.039+0000 7f6bab7fe640 20 client.4289.objecter dump_active .. 0 homeless 2023-10-30T20:24:08.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.039+0000 7f6bab7fe640 1 -- 172.21.15.69:0/883138111 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6bb8079760 con 0x7f6bbc0070a0 2023-10-30T20:24:08.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.039+0000 7f6bab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.039+0000 7f6bab7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:08.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.042+0000 7fa61ffff640 1 -- 172.21.15.69:0/3426682588 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fa6280585b0 con 0x7fa6381513b0 2023-10-30T20:24:08.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.042+0000 7fa61ffff640 10 client.4311.objecter ms_dispatch 0x7fa638000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:08.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.042+0000 7fa61ffff640 3 client.4311.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:08.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.042+0000 7fa61ffff640 3 client.4311.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:08.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.042+0000 7fa61ffff640 20 client.4311.objecter dump_active .. 0 homeless 2023-10-30T20:24:08.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.042+0000 7f148f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14900b1290 0x7f1490134740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45420/0 (socket says 172.21.15.69:45420) 2023-10-30T20:24:08.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.042+0000 7f148f7fe640 1 -- 172.21.15.69:0/2996165042 learned_addr learned my addr 172.21.15.69:0/2996165042 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:08.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.042+0000 7f148f7fe640 10 monclient(hunting): get_auth_request con 0x7f14900b1290 auth_method 0 2023-10-30T20:24:08.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.042+0000 7f148f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:08.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.042+0000 7f148f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:08.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.042+0000 7f148f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:08.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.042+0000 7fa61ffff640 1 -- 172.21.15.69:0/3426682588 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa628057eb0 con 0x7fa6381513b0 2023-10-30T20:24:08.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.042+0000 7fa61ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.042+0000 7fa61ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:08.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.046+0000 7f148f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:08.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.046+0000 7f148f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:08.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.046+0000 7f148f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:08.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.051+0000 7f148f7fe640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2023-10-30T20:24:08.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.051+0000 7f148f7fe640 1 -- 172.21.15.69:0/2996165042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14900ae140 msgr2=0x7f1490134200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.051+0000 7f148f7fe640 1 --2- 172.21.15.69:0/2996165042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14900ae140 0x7f1490134200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.051+0000 7f148f7fe640 1 -- 172.21.15.69:0/2996165042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1490070110 msgr2=0x7f14900a9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.051+0000 7f148f7fe640 1 --2- 172.21.15.69:0/2996165042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1490070110 0x7f14900a9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.051+0000 7f148f7fe640 10 monclient: _finish_hunting 0 2023-10-30T20:24:08.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.051+0000 7f148f7fe640 1 monclient: found mon.noname-a 2023-10-30T20:24:08.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.051+0000 7f148f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:08.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.051+0000 7f148f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:08.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.051+0000 7f148f7fe640 1 -- 172.21.15.69:0/2996165042 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14900abc10 con 0x7f14900b1290 2023-10-30T20:24:08.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.060+0000 7f148f7fe640 1 --2- 172.21.15.69:0/2996165042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14900b1290 0x7f1490134740 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f148800c780 tx=0x7f148800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:08.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.060+0000 7f1487fff640 10 client.?.objecter ms_handle_connect 0x7f14900b1290 2023-10-30T20:24:08.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.060+0000 7f1487fff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:08.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f1487fff640 1 -- 172.21.15.69:0/2996165042 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148800ce90 con 0x7f14900b1290 2023-10-30T20:24:08.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f1487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:08.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f1487fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:08.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f1487fff640 10 monclient: dump: 2023-10-30T20:24:08.068 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:08.068 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:08.068 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.068 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.068 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:08.069 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:08.069 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:08.069 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:08.069 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:08.069 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:08.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f1487fff640 10 monclient: _finish_auth 0 2023-10-30T20:24:08.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f1487fff640 10 monclient: _check_auth_tickets 2023-10-30T20:24:08.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f1487fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:08.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f1487fff640 1 -- 172.21.15.69:0/2996165042 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148800e040 con 0x7f14900b1290 2023-10-30T20:24:08.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f1487fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:08.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f1487fff640 1 -- 172.21.15.69:0/2996165042 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488005070 con 0x7f14900b1290 2023-10-30T20:24:08.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f1487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:08.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f1487fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:08.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f1487fff640 10 monclient: dump: 2023-10-30T20:24:08.071 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:08.071 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:08.071 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.072 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.072 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:08.072 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:08.072 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:08.072 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:08.073 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:08.073 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:08.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f14953d5640 5 monclient: authenticate success, global_id 4326 2023-10-30T20:24:08.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f14953d5640 10 monclient: _renew_subs 2023-10-30T20:24:08.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f148dffb640 4 set_mon_vals no callback set 2023-10-30T20:24:08.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f14953d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:08.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f14953d5640 1 -- 172.21.15.69:0/2996165042 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14900aa5a0 con 0x7f14900b1290 2023-10-30T20:24:08.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f14953d5640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:08.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f14953d5640 10 monclient: _renew_subs 2023-10-30T20:24:08.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f14953d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:08.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f14953d5640 1 -- 172.21.15.69:0/2996165042 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14900a8970 con 0x7f14900b1290 2023-10-30T20:24:08.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.065+0000 7f14953d5640 1 librados: init done 2023-10-30T20:24:08.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.066+0000 7f14953d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.066+0000 7f14953d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.066+0000 7f14953d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:08.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.066+0000 7f14953d5640 1 -- 172.21.15.69:0/2996165042 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14900aa7a0 con 0x7f14900b1290 2023-10-30T20:24:08.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.089+0000 7f1487fff640 1 -- 172.21.15.69:0/2996165042 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f1488006b10 con 0x7f14900b1290 2023-10-30T20:24:08.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.090+0000 7f1487fff640 1 --2- 172.21.15.69:0/2996165042 >> v2:172.21.15.69:6800/1876307353 conn(0x7f147006c7c0 0x7f147006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:08.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.090+0000 7f148effd640 1 --2- 172.21.15.69:0/2996165042 >> v2:172.21.15.69:6800/1876307353 conn(0x7f147006c7c0 0x7f147006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:08.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.090+0000 7f148effd640 10 monclient: get_auth_request con 0x7f147006c7c0 auth_method 0 2023-10-30T20:24:08.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.090+0000 7f148effd640 1 --2- 172.21.15.69:0/2996165042 >> v2:172.21.15.69:6800/1876307353 conn(0x7f147006c7c0 0x7f147006ec80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1480037000 tx=0x7f1480008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:08.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.090+0000 7f1487fff640 10 client.4326.objecter ms_handle_connect 0x7f147006c7c0 2023-10-30T20:24:08.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.218+0000 7f6bc3fff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T20:24:08.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.218+0000 7f6bc3fff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T20:24:08.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.218+0000 7f6bc3fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:08.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.218+0000 7f6bc3fff640 1 -- 172.21.15.69:0/883138111 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f6bbc036540 con 0x7f6bbc0070a0 2023-10-30T20:24:08.225 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.223+0000 7fa63dba9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T20:24:08.225 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.223+0000 7fa63dba9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T20:24:08.225 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.223+0000 7fa63dba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:08.225 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.223+0000 7fa63dba9640 1 -- 172.21.15.69:0/3426682588 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa63813d630 con 0x7fa6381513b0 2023-10-30T20:24:08.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.247+0000 7f7fb77fe640 1 --2- 172.21.15.69:0/314820218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fb813b4d0 0x7f7fb813b950 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0036040 tx=0x7f7fb0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:08.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.247+0000 7f7f9f7fe640 10 client.?.objecter ms_handle_connect 0x7f7fb813b4d0 2023-10-30T20:24:08.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.248+0000 7f7f9f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:08.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.269+0000 7f74f77fe640 1 -- 172.21.15.69:0/3954807367 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f75100d1f30 con 0x7f750815b930 2023-10-30T20:24:08.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.269+0000 7f74f77fe640 10 client.4324.objecter ms_dispatch 0x7f7510000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:08.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.269+0000 7f74f77fe640 3 client.4324.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:08.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.269+0000 7f74f77fe640 3 client.4324.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:08.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.269+0000 7f74f77fe640 20 client.4324.objecter dump_active .. 0 homeless 2023-10-30T20:24:08.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.282+0000 7f74f77fe640 1 -- 172.21.15.69:0/3954807367 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f75100d1830 con 0x7f750815b930 2023-10-30T20:24:08.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.282+0000 7f74f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.282+0000 7f74f77fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:08.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7f9f7fe640 1 -- 172.21.15.69:0/314820218 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb003e450 con 0x7f7fb813b4d0 2023-10-30T20:24:08.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7f9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:08.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7f9f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:08.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7f9f7fe640 10 monclient: dump: 2023-10-30T20:24:08.286 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:08.286 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:08.286 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:08.288 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:08.288 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:08.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7f9f7fe640 10 monclient: _finish_auth 0 2023-10-30T20:24:08.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7f9f7fe640 10 monclient: _check_auth_tickets 2023-10-30T20:24:08.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7f9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:08.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7f9f7fe640 1 -- 172.21.15.69:0/314820218 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb003e5c0 con 0x7f7fb813b4d0 2023-10-30T20:24:08.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7f9f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:08.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7f9f7fe640 1 -- 172.21.15.69:0/314820218 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb003e9c0 con 0x7f7fb813b4d0 2023-10-30T20:24:08.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7fbd390640 5 monclient: authenticate success, global_id 4342 2023-10-30T20:24:08.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7fbd390640 10 monclient: _renew_subs 2023-10-30T20:24:08.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7fbd390640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:08.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7fbd390640 1 -- 172.21.15.69:0/314820218 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb81d6d90 con 0x7f7fb813b4d0 2023-10-30T20:24:08.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7fb57fa640 4 set_mon_vals no callback set 2023-10-30T20:24:08.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.283+0000 7f7f9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:08.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.284+0000 7f7f9f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:08.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.284+0000 7f7f9f7fe640 10 monclient: dump: 2023-10-30T20:24:08.291 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:08.291 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:08.291 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.291 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:08.292 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:08.292 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:08.292 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:08.292 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:08.292 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:08.292 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:08.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.284+0000 7f7fbd390640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:08.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.284+0000 7f7fbd390640 10 monclient: _renew_subs 2023-10-30T20:24:08.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.284+0000 7f7fbd390640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:08.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.284+0000 7f7fbd390640 1 -- 172.21.15.69:0/314820218 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb81d7250 con 0x7f7fb813b4d0 2023-10-30T20:24:08.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.284+0000 7f7fbd390640 1 librados: init done 2023-10-30T20:24:08.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.284+0000 7f7fbd390640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.284+0000 7f7fbd390640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.284+0000 7f7fbd390640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:08.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.284+0000 7f7fbd390640 1 -- 172.21.15.69:0/314820218 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb81409f0 con 0x7f7fb813b4d0 2023-10-30T20:24:08.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.297+0000 7f4eabfff640 1 -- 172.21.15.69:0/3316644440 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f4ebc02c9b0 con 0x7f4ec40b1b00 2023-10-30T20:24:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.297+0000 7f4eabfff640 1 --2- 172.21.15.69:0/3316644440 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4eb006c7c0 0x7f4eb006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.298+0000 7f4ec2ffd640 1 --2- 172.21.15.69:0/3316644440 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4eb006c7c0 0x7f4eb006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.298+0000 7f4ec2ffd640 10 monclient: get_auth_request con 0x7f4eb006c7c0 auth_method 0 2023-10-30T20:24:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.298+0000 7f4eabfff640 10 client.4327.objecter ms_handle_connect 0x7f4eb006c7c0 2023-10-30T20:24:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.298+0000 7f4ec2ffd640 1 --2- 172.21.15.69:0/3316644440 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4eb006c7c0 0x7f4eb006ec80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4142e30 tx=0x7f4eb40079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:08.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.301+0000 7f4eabfff640 1 -- 172.21.15.69:0/3316644440 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f4ebc07b5f0 con 0x7f4ec40b1b00 2023-10-30T20:24:08.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.301+0000 7f4eabfff640 10 client.4327.objecter ms_dispatch 0x7f4ec4000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:08.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.301+0000 7f4eabfff640 3 client.4327.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:08.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.301+0000 7f4eabfff640 3 client.4327.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:08.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.302+0000 7f4eabfff640 20 client.4327.objecter dump_active .. 0 homeless 2023-10-30T20:24:08.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.312+0000 7f4eabfff640 1 -- 172.21.15.69:0/3316644440 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4ebc07aef0 con 0x7f4ec40b1b00 2023-10-30T20:24:08.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.312+0000 7f4eabfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.312+0000 7f4eabfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:08.328 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.326+0000 7f7973fff640 1 -- 172.21.15.69:0/276416905 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f7988092590 con 0x7f79800c3d00 2023-10-30T20:24:08.328 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.327+0000 7f7973fff640 1 --2- 172.21.15.69:0/276416905 >> v2:172.21.15.69:6800/1876307353 conn(0x7f796c06c810 0x7f796c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:08.328 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.327+0000 7f79877fe640 1 --2- 172.21.15.69:0/276416905 >> v2:172.21.15.69:6800/1876307353 conn(0x7f796c06c810 0x7f796c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:08.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.327+0000 7f79877fe640 10 monclient: get_auth_request con 0x7f796c06c810 auth_method 0 2023-10-30T20:24:08.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.327+0000 7f7973fff640 10 client.4292.objecter ms_handle_connect 0x7f796c06c810 2023-10-30T20:24:08.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.327+0000 7f79877fe640 1 --2- 172.21.15.69:0/276416905 >> v2:172.21.15.69:6800/1876307353 conn(0x7f796c06c810 0x7f796c06ecd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f797c006fd0 tx=0x7f797c009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:08.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.361+0000 7f1487fff640 1 -- 172.21.15.69:0/2996165042 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f1488057a00 con 0x7f14900b1290 2023-10-30T20:24:08.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.361+0000 7f1487fff640 10 client.4326.objecter ms_dispatch 0x7f1490000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:08.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.361+0000 7f1487fff640 3 client.4326.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:08.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.361+0000 7f1487fff640 3 client.4326.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:08.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.361+0000 7f1487fff640 20 client.4326.objecter dump_active .. 0 homeless 2023-10-30T20:24:08.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.362+0000 7f1487fff640 1 -- 172.21.15.69:0/2996165042 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1488057300 con 0x7f14900b1290 2023-10-30T20:24:08.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.362+0000 7f1487fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.362+0000 7f1487fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:08.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.458+0000 7fa61ffff640 1 -- 172.21.15.69:0/3426682588 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa62805e0d0 con 0x7fa6381513b0 2023-10-30T20:24:08.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.458+0000 7fa61ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T20:24:08.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.458+0000 7fa61ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:08.460 INFO:teuthology.orchestra.run.smithi069.stdout:68719476740 2023-10-30T20:24:08.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 10 librados: watch_flush enter 2023-10-30T20:24:08.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 10 librados: watch_flush exit 2023-10-30T20:24:08.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 20 client.4311.objecter shutdown clearing up homeless session... 2023-10-30T20:24:08.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 10 client.4311.objecter successfully canceled tick 2023-10-30T20:24:08.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 1 -- 172.21.15.69:0/3426682588 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa624075020 msgr2=0x7fa6240774e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.464 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 1 --2- 172.21.15.69:0/3426682588 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa624075020 0x7fa6240774e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa62c034000 tx=0x7fa62c02f000 comp rx=0 tx=0).stop 2023-10-30T20:24:08.464 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 10 monclient: shutdown 2023-10-30T20:24:08.464 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:08.464 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 1 -- 172.21.15.69:0/3426682588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6381513b0 msgr2=0x7fa6380aae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 1 --2- 172.21.15.69:0/3426682588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6381513b0 0x7fa6380aae50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa62800c5e0 tx=0x7fa62800caa0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 1 -- 172.21.15.69:0/3426682588 shutdown_connections 2023-10-30T20:24:08.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 1 --2- 172.21.15.69:0/3426682588 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa624075020 0x7fa6240774e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 1 --2- 172.21.15.69:0/3426682588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6381521d0 0x7fa63813d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 1 --2- 172.21.15.69:0/3426682588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa638151df0 0x7fa63813de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.466 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 1 --2- 172.21.15.69:0/3426682588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6381513b0 0x7fa6380aae50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.466 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 1 -- 172.21.15.69:0/3426682588 >> 172.21.15.69:0/3426682588 conn(0x7fa6380afba0 msgr2=0x7fa6380ae140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:08.466 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 1 -- 172.21.15.69:0/3426682588 shutdown_connections 2023-10-30T20:24:08.466 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 1 -- 172.21.15.69:0/3426682588 wait complete. 2023-10-30T20:24:08.466 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.461+0000 7fa63dba9640 1 librados: shutdown 2023-10-30T20:24:08.478 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.1 2023-10-30T20:24:08.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.522+0000 7f4ec99a9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T20:24:08.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.522+0000 7f4ec99a9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T20:24:08.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.522+0000 7f4ec99a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:08.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.522+0000 7f4ec99a9640 1 -- 172.21.15.69:0/3316644440 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f4ec40b06a0 con 0x7f4ec40b1b00 2023-10-30T20:24:08.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.533+0000 7f75149ed640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T20:24:08.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.533+0000 7f75149ed640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T20:24:08.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.533+0000 7f75149ed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:08.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.533+0000 7f75149ed640 1 -- 172.21.15.69:0/3954807367 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f750802fe50 con 0x7f750815b930 2023-10-30T20:24:08.538 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.537+0000 7f7973fff640 1 -- 172.21.15.69:0/276416905 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f79880d1f90 con 0x7f79800c3d00 2023-10-30T20:24:08.538 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.537+0000 7f7973fff640 10 client.4292.objecter ms_dispatch 0x7f7988000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:08.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.537+0000 7f7973fff640 3 client.4292.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:08.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.537+0000 7f7973fff640 3 client.4292.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:08.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.537+0000 7f7973fff640 20 client.4292.objecter dump_active .. 0 homeless 2023-10-30T20:24:08.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.560+0000 7f14953d5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T20:24:08.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.560+0000 7f14953d5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T20:24:08.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.560+0000 7f14953d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:08.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.560+0000 7f14953d5640 1 -- 172.21.15.69:0/2996165042 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f149004e9b0 con 0x7f14900b1290 2023-10-30T20:24:08.565 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.564+0000 7f1487fff640 1 -- 172.21.15.69:0/2996165042 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f148805b450 con 0x7f14900b1290 2023-10-30T20:24:08.565 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.564+0000 7f1487fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T20:24:08.565 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.564+0000 7f1487fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:08.566 INFO:teuthology.orchestra.run.smithi069.stdout:55834574853 2023-10-30T20:24:08.567 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.565+0000 7f14953d5640 10 librados: watch_flush enter 2023-10-30T20:24:08.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.565+0000 7f14953d5640 10 librados: watch_flush exit 2023-10-30T20:24:08.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.565+0000 7f14953d5640 20 client.4326.objecter shutdown clearing up homeless session... 2023-10-30T20:24:08.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.565+0000 7f14953d5640 10 client.4326.objecter successfully canceled tick 2023-10-30T20:24:08.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.565+0000 7f14953d5640 1 -- 172.21.15.69:0/2996165042 >> v2:172.21.15.69:6800/1876307353 conn(0x7f147006c7c0 msgr2=0x7f147006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.565+0000 7f14953d5640 1 --2- 172.21.15.69:0/2996165042 >> v2:172.21.15.69:6800/1876307353 conn(0x7f147006c7c0 0x7f147006ec80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1480037000 tx=0x7f1480008040 comp rx=0 tx=0).stop 2023-10-30T20:24:08.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.565+0000 7f14953d5640 10 monclient: shutdown 2023-10-30T20:24:08.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.565+0000 7f14953d5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:08.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.565+0000 7f14953d5640 1 -- 172.21.15.69:0/2996165042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14900b1290 msgr2=0x7f1490134740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.565+0000 7f14953d5640 1 --2- 172.21.15.69:0/2996165042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14900b1290 0x7f1490134740 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f148800c780 tx=0x7f148800cc40 comp rx=0 tx=0).stop 2023-10-30T20:24:08.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.566+0000 7f14953d5640 1 -- 172.21.15.69:0/2996165042 shutdown_connections 2023-10-30T20:24:08.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.566+0000 7f14953d5640 1 --2- 172.21.15.69:0/2996165042 >> v2:172.21.15.69:6800/1876307353 conn(0x7f147006c7c0 0x7f147006ec80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.566+0000 7f14953d5640 1 --2- 172.21.15.69:0/2996165042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14900b1290 0x7f1490134740 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.566+0000 7f14953d5640 1 --2- 172.21.15.69:0/2996165042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14900ae140 0x7f1490134200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.566+0000 7f14953d5640 1 --2- 172.21.15.69:0/2996165042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1490070110 0x7f14900a9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.566+0000 7f14953d5640 1 -- 172.21.15.69:0/2996165042 >> 172.21.15.69:0/2996165042 conn(0x7f14900b4460 msgr2=0x7f149013a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:08.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.566+0000 7f14953d5640 1 -- 172.21.15.69:0/2996165042 shutdown_connections 2023-10-30T20:24:08.571 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.566+0000 7f14953d5640 1 -- 172.21.15.69:0/2996165042 wait complete. 2023-10-30T20:24:08.571 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.566+0000 7f14953d5640 1 librados: shutdown 2023-10-30T20:24:08.576 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.575+0000 7f7f9f7fe640 1 -- 172.21.15.69:0/314820218 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f7fb002ca00 con 0x7f7fb813b4d0 2023-10-30T20:24:08.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.576+0000 7f7f9f7fe640 1 --2- 172.21.15.69:0/314820218 >> v2:172.21.15.69:6800/1876307353 conn(0x7f7fa406c810 0x7f7fa406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:08.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.576+0000 7f7fb6ffd640 1 --2- 172.21.15.69:0/314820218 >> v2:172.21.15.69:6800/1876307353 conn(0x7f7fa406c810 0x7f7fa406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.576+0000 7f7fb6ffd640 10 monclient: get_auth_request con 0x7f7fa406c810 auth_method 0 2023-10-30T20:24:08.579 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2023-10-30T20:24:08.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.576+0000 7f7fb6ffd640 1 --2- 172.21.15.69:0/314820218 >> v2:172.21.15.69:6800/1876307353 conn(0x7f7fa406c810 0x7f7fa406ecd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009890 tx=0x7f7fa8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:08.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.576+0000 7f7f9f7fe640 10 client.4342.objecter ms_handle_connect 0x7f7fa406c810 2023-10-30T20:24:08.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.587+0000 7f7f9f7fe640 1 -- 172.21.15.69:0/314820218 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f7fb00333a0 con 0x7f7fb813b4d0 2023-10-30T20:24:08.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.587+0000 7f7f9f7fe640 10 client.4342.objecter ms_dispatch 0x7f7fb8000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:08.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.587+0000 7f7f9f7fe640 3 client.4342.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:08.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.587+0000 7f7f9f7fe640 3 client.4342.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:08.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.587+0000 7f7f9f7fe640 20 client.4342.objecter dump_active .. 0 homeless 2023-10-30T20:24:08.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.596+0000 7f7f9f7fe640 1 -- 172.21.15.69:0/314820218 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7fb007b520 con 0x7f7fb813b4d0 2023-10-30T20:24:08.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.596+0000 7f7f9f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.596+0000 7f7f9f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:08.608 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.607+0000 7f4eabfff640 1 -- 172.21.15.69:0/3316644440 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4ebc07f040 con 0x7f4ec40b1b00 2023-10-30T20:24:08.608 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.607+0000 7f4eabfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T20:24:08.608 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.607+0000 7f4eabfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:08.609 INFO:teuthology.orchestra.run.smithi069.stdout:60129542148 2023-10-30T20:24:08.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.607+0000 7f74f77fe640 1 -- 172.21.15.69:0/3954807367 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f75100d5980 con 0x7f750815b930 2023-10-30T20:24:08.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.607+0000 7f74f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T20:24:08.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.607+0000 7f74f77fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:08.610 INFO:teuthology.orchestra.run.smithi069.stdout:64424509445 2023-10-30T20:24:08.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 10 librados: watch_flush enter 2023-10-30T20:24:08.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 10 librados: watch_flush exit 2023-10-30T20:24:08.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 20 client.4327.objecter shutdown clearing up homeless session... 2023-10-30T20:24:08.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 10 client.4327.objecter successfully canceled tick 2023-10-30T20:24:08.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 1 -- 172.21.15.69:0/3316644440 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4eb006c7c0 msgr2=0x7f4eb006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 1 --2- 172.21.15.69:0/3316644440 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4eb006c7c0 0x7f4eb006ec80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4142e30 tx=0x7f4eb40079b0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 10 monclient: shutdown 2023-10-30T20:24:08.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:08.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 1 -- 172.21.15.69:0/3316644440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ec40b1b00 msgr2=0x7f4ec412f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 1 --2- 172.21.15.69:0/3316644440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ec40b1b00 0x7f4ec412f6a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc036040 tx=0x7f4ebc004a40 comp rx=0 tx=0).stop 2023-10-30T20:24:08.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 1 -- 172.21.15.69:0/3316644440 shutdown_connections 2023-10-30T20:24:08.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 1 --2- 172.21.15.69:0/3316644440 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4eb006c7c0 0x7f4eb006ec80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 1 --2- 172.21.15.69:0/3316644440 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec4143220 0x7f4ec40b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 1 --2- 172.21.15.69:0/3316644440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ec40b1b00 0x7f4ec412f6a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 1 --2- 172.21.15.69:0/3316644440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec40a88d0 0x7f4ec40ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 1 -- 172.21.15.69:0/3316644440 >> 172.21.15.69:0/3316644440 conn(0x7f4ec40b7180 msgr2=0x7f4ec41422a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:08.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 1 -- 172.21.15.69:0/3316644440 shutdown_connections 2023-10-30T20:24:08.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 1 -- 172.21.15.69:0/3316644440 wait complete. 2023-10-30T20:24:08.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f4ec99a9640 1 librados: shutdown 2023-10-30T20:24:08.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f75149ed640 10 librados: watch_flush enter 2023-10-30T20:24:08.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f75149ed640 10 librados: watch_flush exit 2023-10-30T20:24:08.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.609+0000 7f75149ed640 20 client.4324.objecter shutdown clearing up homeless session... 2023-10-30T20:24:08.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 10 client.4324.objecter successfully canceled tick 2023-10-30T20:24:08.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 1 -- 172.21.15.69:0/3954807367 >> v2:172.21.15.69:6800/1876307353 conn(0x7f74fc06c810 msgr2=0x7f74fc06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 1 --2- 172.21.15.69:0/3954807367 >> v2:172.21.15.69:6800/1876307353 conn(0x7f74fc06c810 0x7f74fc06ecd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7500009670 tx=0x7f7500006bf0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 10 monclient: shutdown 2023-10-30T20:24:08.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:08.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 1 -- 172.21.15.69:0/3954807367 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f750815b930 msgr2=0x7f750815bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 1 --2- 172.21.15.69:0/3954807367 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f750815b930 0x7f750815bdb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7510066dc0 tx=0x7f7510067340 comp rx=0 tx=0).stop 2023-10-30T20:24:08.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 1 -- 172.21.15.69:0/3954807367 shutdown_connections 2023-10-30T20:24:08.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 1 --2- 172.21.15.69:0/3954807367 >> v2:172.21.15.69:6800/1876307353 conn(0x7f74fc06c810 0x7f74fc06ecd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 1 --2- 172.21.15.69:0/3954807367 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f750815b930 0x7f750815bdb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 1 --2- 172.21.15.69:0/3954807367 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7508037670 0x7f750802f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 1 --2- 172.21.15.69:0/3954807367 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f750802b200 0x7f750802f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 1 -- 172.21.15.69:0/3954807367 >> 172.21.15.69:0/3954807367 conn(0x7f750803a9e0 msgr2=0x7f7508006a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:08.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 1 -- 172.21.15.69:0/3954807367 shutdown_connections 2023-10-30T20:24:08.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 1 -- 172.21.15.69:0/3954807367 wait complete. 2023-10-30T20:24:08.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.610+0000 7f75149ed640 1 librados: shutdown 2023-10-30T20:24:08.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.612+0000 7f7973fff640 1 -- 172.21.15.69:0/276416905 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f79880d1890 con 0x7f79800c3d00 2023-10-30T20:24:08.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.612+0000 7f7973fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.612+0000 7f7973fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:08.624 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.6 2023-10-30T20:24:08.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.2 2023-10-30T20:24:08.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.634+0000 7fe327fff640 1 -- 172.21.15.69:0/298708272 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fe33801e440 con 0x7fe3401da4d0 2023-10-30T20:24:08.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.635+0000 7fe327fff640 1 --2- 172.21.15.69:0/298708272 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe32c06c7f0 0x7fe32c06ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:08.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.635+0000 7fe33f7fe640 1 --2- 172.21.15.69:0/298708272 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe32c06c7f0 0x7fe32c06ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:08.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.635+0000 7fe33f7fe640 10 monclient: get_auth_request con 0x7fe32c06c7f0 auth_method 0 2023-10-30T20:24:08.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.635+0000 7fe33f7fe640 1 --2- 172.21.15.69:0/298708272 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe32c06c7f0 0x7fe32c06ecb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe330035000 tx=0x7fe330008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:08.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.635+0000 7fe327fff640 10 client.4295.objecter ms_handle_connect 0x7fe32c06c7f0 2023-10-30T20:24:08.649 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.648+0000 7fe327fff640 1 -- 172.21.15.69:0/298708272 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fe33800ce30 con 0x7fe3401da4d0 2023-10-30T20:24:08.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.648+0000 7fe327fff640 10 client.4295.objecter ms_dispatch 0x7fe340000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:08.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.648+0000 7fe327fff640 3 client.4295.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:08.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.648+0000 7fe327fff640 3 client.4295.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:08.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.648+0000 7fe327fff640 20 client.4295.objecter dump_active .. 0 homeless 2023-10-30T20:24:08.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.648+0000 7fe327fff640 1 -- 172.21.15.69:0/298708272 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe3380581a0 con 0x7fe3401da4d0 2023-10-30T20:24:08.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.648+0000 7fe327fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:08.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.648+0000 7fe327fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:08.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.668+0000 7f6bab7fe640 1 -- 172.21.15.69:0/883138111 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6bb807d8b0 con 0x7f6bbc0070a0 2023-10-30T20:24:08.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.668+0000 7f6bab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T20:24:08.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.668+0000 7f6bab7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:08.671 INFO:teuthology.orchestra.run.smithi069.stdout:60129542147 2023-10-30T20:24:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 10 librados: watch_flush enter 2023-10-30T20:24:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 10 librados: watch_flush exit 2023-10-30T20:24:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 20 client.4289.objecter shutdown clearing up homeless session... 2023-10-30T20:24:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 10 client.4289.objecter successfully canceled tick 2023-10-30T20:24:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 1 -- 172.21.15.69:0/883138111 >> v2:172.21.15.69:6800/1876307353 conn(0x7f6bb006c7c0 msgr2=0x7f6bb006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 1 --2- 172.21.15.69:0/883138111 >> v2:172.21.15.69:6800/1876307353 conn(0x7f6bb006c7c0 0x7f6bb006ec80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc0328a0 tx=0x7f6bb4008040 comp rx=0 tx=0).stop 2023-10-30T20:24:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 10 monclient: shutdown 2023-10-30T20:24:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 1 -- 172.21.15.69:0/883138111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bbc0070a0 msgr2=0x7f6bbc0cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 1 --2- 172.21.15.69:0/883138111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bbc0070a0 0x7f6bbc0cbd60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8000c00 tx=0x7f6bb80046c0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 1 -- 172.21.15.69:0/883138111 shutdown_connections 2023-10-30T20:24:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 1 --2- 172.21.15.69:0/883138111 >> v2:172.21.15.69:6800/1876307353 conn(0x7f6bb006c7c0 0x7f6bb006ec80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 1 --2- 172.21.15.69:0/883138111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6bbc0ce8d0 0x7f6bbc0cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 1 --2- 172.21.15.69:0/883138111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bbc0360b0 0x7f6bbc0320f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.671+0000 7f6bc3fff640 1 --2- 172.21.15.69:0/883138111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bbc0070a0 0x7f6bbc0cbd60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.672+0000 7f6bc3fff640 1 -- 172.21.15.69:0/883138111 >> 172.21.15.69:0/883138111 conn(0x7f6bbc006040 msgr2=0x7f6bbc0bdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.672+0000 7f6bc3fff640 1 -- 172.21.15.69:0/883138111 shutdown_connections 2023-10-30T20:24:08.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.672+0000 7f6bc3fff640 1 -- 172.21.15.69:0/883138111 wait complete. 2023-10-30T20:24:08.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.672+0000 7f6bc3fff640 1 librados: shutdown 2023-10-30T20:24:08.689 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.7 2023-10-30T20:24:08.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.797+0000 7f7fbd390640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T20:24:08.799 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.797+0000 7f7fbd390640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T20:24:08.799 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.797+0000 7f7fbd390640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:08.799 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.797+0000 7f7fbd390640 1 -- 172.21.15.69:0/314820218 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7fb80a8920 con 0x7f7fb813b4d0 2023-10-30T20:24:08.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.801+0000 7f7f9f7fe640 1 -- 172.21.15.69:0/314820218 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7fb007ae20 con 0x7f7fb813b4d0 2023-10-30T20:24:08.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.801+0000 7f7f9f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T20:24:08.803 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.801+0000 7f7f9f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:08.803 INFO:teuthology.orchestra.run.smithi069.stdout:68719476742 2023-10-30T20:24:08.805 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.803+0000 7f7fbd390640 10 librados: watch_flush enter 2023-10-30T20:24:08.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.803+0000 7f7fbd390640 10 librados: watch_flush exit 2023-10-30T20:24:08.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.803+0000 7f7fbd390640 20 client.4342.objecter shutdown clearing up homeless session... 2023-10-30T20:24:08.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.803+0000 7f7fbd390640 10 client.4342.objecter successfully canceled tick 2023-10-30T20:24:08.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 1 -- 172.21.15.69:0/314820218 >> v2:172.21.15.69:6800/1876307353 conn(0x7f7fa406c810 msgr2=0x7f7fa406ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 1 --2- 172.21.15.69:0/314820218 >> v2:172.21.15.69:6800/1876307353 conn(0x7f7fa406c810 0x7f7fa406ecd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009890 tx=0x7f7fa8008040 comp rx=0 tx=0).stop 2023-10-30T20:24:08.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 10 monclient: shutdown 2023-10-30T20:24:08.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:08.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 1 -- 172.21.15.69:0/314820218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fb813b4d0 msgr2=0x7f7fb813b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 1 --2- 172.21.15.69:0/314820218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fb813b4d0 0x7f7fb813b950 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0036040 tx=0x7f7fb0004a40 comp rx=0 tx=0).stop 2023-10-30T20:24:08.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 1 -- 172.21.15.69:0/314820218 shutdown_connections 2023-10-30T20:24:08.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 1 --2- 172.21.15.69:0/314820218 >> v2:172.21.15.69:6800/1876307353 conn(0x7f7fa406c810 0x7f7fa406ecd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 1 --2- 172.21.15.69:0/314820218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fb813b4d0 0x7f7fb813b950 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 1 --2- 172.21.15.69:0/314820218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fb813faa0 0x7f7fb81404b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 1 --2- 172.21.15.69:0/314820218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fb80a8e40 0x7f7fb813ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 1 -- 172.21.15.69:0/314820218 >> 172.21.15.69:0/314820218 conn(0x7f7fb80b7180 msgr2=0x7f7fb81cc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:08.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 1 -- 172.21.15.69:0/314820218 shutdown_connections 2023-10-30T20:24:08.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 1 -- 172.21.15.69:0/314820218 wait complete. 2023-10-30T20:24:08.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.804+0000 7f7fbd390640 1 librados: shutdown 2023-10-30T20:24:08.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.807+0000 7f798cd67640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T20:24:08.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.807+0000 7f798cd67640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T20:24:08.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.807+0000 7f798cd67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:08.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.807+0000 7f798cd67640 1 -- 172.21.15.69:0/276416905 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f7980160130 con 0x7f79800c3d00 2023-10-30T20:24:08.814 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.813+0000 7f7973fff640 1 -- 172.21.15.69:0/276416905 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f79880d59e0 con 0x7f79800c3d00 2023-10-30T20:24:08.814 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.813+0000 7f7973fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T20:24:08.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.813+0000 7f7973fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:08.815 INFO:teuthology.orchestra.run.smithi069.stdout:64424509443 2023-10-30T20:24:08.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 10 librados: watch_flush enter 2023-10-30T20:24:08.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 10 librados: watch_flush exit 2023-10-30T20:24:08.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 20 client.4292.objecter shutdown clearing up homeless session... 2023-10-30T20:24:08.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 10 client.4292.objecter successfully canceled tick 2023-10-30T20:24:08.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 1 -- 172.21.15.69:0/276416905 >> v2:172.21.15.69:6800/1876307353 conn(0x7f796c06c810 msgr2=0x7f796c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 1 --2- 172.21.15.69:0/276416905 >> v2:172.21.15.69:6800/1876307353 conn(0x7f796c06c810 0x7f796c06ecd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f797c006fd0 tx=0x7f797c009060 comp rx=0 tx=0).stop 2023-10-30T20:24:08.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 10 monclient: shutdown 2023-10-30T20:24:08.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:08.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 1 -- 172.21.15.69:0/276416905 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79800c3d00 msgr2=0x7f79800c72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 1 --2- 172.21.15.69:0/276416905 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79800c3d00 0x7f79800c72f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7988066dc0 tx=0x7f7988067340 comp rx=0 tx=0).stop 2023-10-30T20:24:08.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 1 -- 172.21.15.69:0/276416905 shutdown_connections 2023-10-30T20:24:08.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 1 --2- 172.21.15.69:0/276416905 >> v2:172.21.15.69:6800/1876307353 conn(0x7f796c06c810 0x7f796c06ecd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 1 --2- 172.21.15.69:0/276416905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79800bb260 0x7f79800bb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 1 --2- 172.21.15.69:0/276416905 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79800c3d00 0x7f79800c72f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 1 --2- 172.21.15.69:0/276416905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7980034aa0 0x7f7980031870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 1 -- 172.21.15.69:0/276416905 >> 172.21.15.69:0/276416905 conn(0x7f79800060a0 msgr2=0x7f798002b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:08.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 1 -- 172.21.15.69:0/276416905 shutdown_connections 2023-10-30T20:24:08.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 1 -- 172.21.15.69:0/276416905 wait complete. 2023-10-30T20:24:08.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.815+0000 7f798cd67640 1 librados: shutdown 2023-10-30T20:24:08.821 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476742 for osd.0 2023-10-30T20:24:08.830 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509443 for osd.5 2023-10-30T20:24:08.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.881+0000 7fe345fe9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T20:24:08.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.881+0000 7fe345fe9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T20:24:08.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.881+0000 7fe345fe9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:08.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.881+0000 7fe345fe9640 1 -- 172.21.15.69:0/298708272 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe340141170 con 0x7fe3401da4d0 2023-10-30T20:24:08.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.885+0000 7fe327fff640 1 -- 172.21.15.69:0/298708272 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe33805c240 con 0x7fe3401da4d0 2023-10-30T20:24:08.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.885+0000 7fe327fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T20:24:08.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.885+0000 7fe327fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:08.887 INFO:teuthology.orchestra.run.smithi069.stdout:68719476742 2023-10-30T20:24:08.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 10 librados: watch_flush enter 2023-10-30T20:24:08.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 10 librados: watch_flush exit 2023-10-30T20:24:08.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 20 client.4295.objecter shutdown clearing up homeless session... 2023-10-30T20:24:08.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 10 client.4295.objecter successfully canceled tick 2023-10-30T20:24:08.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 1 -- 172.21.15.69:0/298708272 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe32c06c7f0 msgr2=0x7fe32c06ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 1 --2- 172.21.15.69:0/298708272 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe32c06c7f0 0x7fe32c06ecb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe330035000 tx=0x7fe330008030 comp rx=0 tx=0).stop 2023-10-30T20:24:08.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 10 monclient: shutdown 2023-10-30T20:24:08.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:08.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 1 -- 172.21.15.69:0/298708272 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3401da4d0 msgr2=0x7fe340136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:08.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 1 --2- 172.21.15.69:0/298708272 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3401da4d0 0x7fe340136f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe33800ba10 tx=0x7fe33800bed0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 1 -- 172.21.15.69:0/298708272 shutdown_connections 2023-10-30T20:24:08.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 1 --2- 172.21.15.69:0/298708272 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe32c06c7f0 0x7fe32c06ecb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 1 --2- 172.21.15.69:0/298708272 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3401da4d0 0x7fe340136f40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 1 --2- 172.21.15.69:0/298708272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3400aacb0 0x7fe3401d9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 1 --2- 172.21.15.69:0/298708272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3400a8e40 0x7fe34013c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:08.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 1 -- 172.21.15.69:0/298708272 >> 172.21.15.69:0/298708272 conn(0x7fe3400b7180 msgr2=0x7fe34013d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:08.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 1 -- 172.21.15.69:0/298708272 shutdown_connections 2023-10-30T20:24:08.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 1 -- 172.21.15.69:0/298708272 wait complete. 2023-10-30T20:24:08.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:08.887+0000 7fe345fe9640 1 librados: shutdown 2023-10-30T20:24:08.900 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.3 2023-10-30T20:24:08.900 DEBUG:teuthology.parallel:result is None 2023-10-30T20:24:09.479 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T20:24:09.580 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T20:24:09.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.584+0000 7f8c108af640 10 monclient: get_monmap_and_config 2023-10-30T20:24:09.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.584+0000 7f8c108af640 10 monclient: build_initial_monmap 2023-10-30T20:24:09.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.584+0000 7f8c108af640 10 monclient: monmap: 2023-10-30T20:24:09.587 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:09.588 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.588 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:09.586683+0000 2023-10-30T20:24:09.588 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:09.586683+0000 2023-10-30T20:24:09.588 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:09.589 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.589 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:09.589 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:09.589 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:09.589 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:09.590 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:09.590 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.585+0000 7f8c108af640 1 Processor -- start 2023-10-30T20:24:09.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.585+0000 7f8c108af640 1 -- start start 2023-10-30T20:24:09.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.585+0000 7f8c108af640 10 monclient: init 2023-10-30T20:24:09.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c108af640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:09.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c108af640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-10-30T20:24:09.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c108af640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0c0a8e40 0x7f8c0c13efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c108af640 10 monclient(hunting): picked mon.noname-b con 0x7f8c0c0a8e40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:09.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c108af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8c0c0b13c0 addr v1:172.21.15.69:6789/0 2023-10-30T20:24:09.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c108af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8c0c13f500 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:09.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c108af640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c0b1b00 con 0x7f8c0c0b13c0 2023-10-30T20:24:09.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c108af640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c0a8920 con 0x7f8c0c13f500 2023-10-30T20:24:09.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c108af640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c108af640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c0679a0 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c108af640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c108af640 10 monclient(hunting): authenticate will time out at 1572.875958s 2023-10-30T20:24:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c0a59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0c0a8e40 0x7f8c0c13efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c03fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8c0c0b13c0 0x7f8c0c141eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36842/0 (socket says 172.21.15.69:36842) 2023-10-30T20:24:09.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.586+0000 7f8c03fff640 1 -- 172.21.15.69:0/3425269001 learned_addr learned my addr 172.21.15.69:0/3425269001 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:24:09.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.587+0000 7f8c0a59c640 10 monclient(hunting): get_auth_request con 0x7f8c0c0a8e40 auth_method 0 2023-10-30T20:24:09.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.587+0000 7f8c0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:09.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.587+0000 7f8c0a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.587+0000 7f8c0a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:09.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.587+0000 7f8c0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:09.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.587+0000 7f8c0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:09.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.587+0000 7f8c0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:09.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c09d9b640 1 -- 172.21.15.69:0/3425269001 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428963736 0 0) 0x7f8c04001490 con 0x7f8c0c13f500 2023-10-30T20:24:09.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c09d9b640 10 monclient(hunting): _negotiate 2023-10-30T20:24:09.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c09d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c09d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:09.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c09d9b640 10 monclient(hunting): my global_id is 4345 2023-10-30T20:24:09.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c09d9b640 1 -- 172.21.15.69:0/3425269001 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bf8003520 con 0x7f8c0c13f500 2023-10-30T20:24:09.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c09d9b640 1 -- 172.21.15.69:0/3425269001 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964956904 0 0) 0x7f8bf4001110 con 0x7f8c0c0b13c0 2023-10-30T20:24:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c09d9b640 10 monclient(hunting): _negotiate 2023-10-30T20:24:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c09d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c09d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c09d9b640 10 monclient(hunting): my global_id is 4329 2023-10-30T20:24:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c09d9b640 1 -- 172.21.15.69:0/3425269001 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bf8003840 con 0x7f8c0c0b13c0 2023-10-30T20:24:09.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c0a59c640 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2023-10-30T20:24:09.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c0a59c640 1 -- 172.21.15.69:0/3425269001 >> v1:172.21.15.183:6789/0 conn(0x7f8c0c13f500 legacy=0x7f8c0c14c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c0a59c640 1 -- 172.21.15.69:0/3425269001 >> v1:172.21.15.69:6789/0 conn(0x7f8c0c0b13c0 legacy=0x7f8c0c141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c0a59c640 10 monclient: _finish_hunting 0 2023-10-30T20:24:09.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c0a59c640 1 monclient: found mon.noname-b 2023-10-30T20:24:09.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c0a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:09.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c0a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:24:09.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.589+0000 7f8c0a59c640 1 -- 172.21.15.69:0/3425269001 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c0a97b0 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.591+0000 7f8c0a59c640 1 --2- 172.21.15.69:0/3425269001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0c0a8e40 0x7f8c0c13efc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0097d0 tx=0x7f8bfc02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=421b22777e7c6773 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:09.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c09d9b640 1 -- 172.21.15.69:0/3425269001 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc031770 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c09d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c09d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:09.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c09d9b640 10 monclient: dump: 2023-10-30T20:24:09.606 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.607 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.607 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.607 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.607 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.607 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.608 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.608 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.608 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.608 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.608 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c09d9b640 10 monclient: _finish_auth 0 2023-10-30T20:24:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c09d9b640 10 monclient: _check_auth_tickets 2023-10-30T20:24:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c09d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c09d9b640 1 -- 172.21.15.69:0/3425269001 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bfc0318e0 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c108af640 5 monclient: authenticate success, global_id 4301 2023-10-30T20:24:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c108af640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:09.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c09d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:09.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c108af640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:09.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c108af640 4 set_mon_vals no callback set 2023-10-30T20:24:09.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c09d9b640 1 -- 172.21.15.69:0/3425269001 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc031c80 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c09d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c09d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:09.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.595+0000 7f8c09d9b640 10 monclient: dump: 2023-10-30T20:24:09.611 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.611 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.611 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.611 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.612 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.612 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.612 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.612 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.612 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.613 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.596+0000 7f8c108af640 10 monclient: shutdown 2023-10-30T20:24:09.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.596+0000 7f8c108af640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:09.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.596+0000 7f8c108af640 1 -- 172.21.15.69:0/3425269001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0c0a8e40 msgr2=0x7f8c0c13efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.596+0000 7f8c108af640 1 --2- 172.21.15.69:0/3425269001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0c0a8e40 0x7f8c0c13efc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0097d0 tx=0x7f8bfc02b810 comp rx=0 tx=0).stop 2023-10-30T20:24:09.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.596+0000 7f8c108af640 1 -- 172.21.15.69:0/3425269001 shutdown_connections 2023-10-30T20:24:09.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.596+0000 7f8c108af640 1 --2- 172.21.15.69:0/3425269001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0c0a8e40 0x7f8c0c13efc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.596+0000 7f8c108af640 1 -- 172.21.15.69:0/3425269001 >> 172.21.15.69:0/3425269001 conn(0x7f8c0c0b7180 msgr2=0x7f8c0c0b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:09.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.596+0000 7f8c108af640 1 -- 172.21.15.69:0/3425269001 shutdown_connections 2023-10-30T20:24:09.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.596+0000 7f8c108af640 1 -- 172.21.15.69:0/3425269001 wait complete. 2023-10-30T20:24:09.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.597+0000 7f8c108af640 10 monclient: build_initial_monmap 2023-10-30T20:24:09.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.597+0000 7f8c108af640 10 monclient: monmap: 2023-10-30T20:24:09.615 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:09.615 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:09.615 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:09.616 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:09.616 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:09.616 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:09.616 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:09.616 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:09.616 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:09.617 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.597+0000 7f8c108af640 1 librados: starting msgr at 2023-10-30T20:24:09.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.597+0000 7f8c108af640 1 librados: starting objecter 2023-10-30T20:24:09.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.597+0000 7f8c108af640 1 Processor -- start 2023-10-30T20:24:09.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 1 -- start start 2023-10-30T20:24:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 1 librados: setting wanted keys 2023-10-30T20:24:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 1 librados: calling monclient init 2023-10-30T20:24:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 10 monclient: init 2023-10-30T20:24:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:24:09.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c0c0a8e40 0x7f8c0c0ad930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 10 monclient(hunting): picked mon.noname-b con 0x7f8c0c0a8e40 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:09.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0c1da150 0x7f8c0c1432d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 10 monclient(hunting): picked mon.noname-c con 0x7f8c0c1da150 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:09.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0c145770 0x7f8c0c1d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 10 monclient(hunting): picked mon.noname-a con 0x7f8c0c145770 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:09.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c0b07c0 con 0x7f8c0c145770 2023-10-30T20:24:09.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.598+0000 7f8c108af640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c145020 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.599+0000 7f8c108af640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.599+0000 7f8c108af640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c1da590 con 0x7f8c0c1da150 2023-10-30T20:24:09.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.599+0000 7f8c108af640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:09.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.599+0000 7f8c108af640 10 monclient(hunting): authenticate will time out at 1572.888834s 2023-10-30T20:24:09.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.599+0000 7f8c0a59c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c0c0a8e40 0x7f8c0c0ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.599+0000 7f8c03fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0c1da150 0x7f8c0c1432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.599+0000 7f8c0a59c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c0c0a8e40 0x7f8c0c0ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:53098/0 (socket says 172.21.15.69:53098) 2023-10-30T20:24:09.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.599+0000 7f8c0a59c640 1 -- 172.21.15.69:0/3851739655 learned_addr learned my addr 172.21.15.69:0/3851739655 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:09.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.599+0000 7f8c0a59c640 10 monclient(hunting): get_auth_request con 0x7f8c0c0a8e40 auth_method 0 2023-10-30T20:24:09.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.599+0000 7f8c0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:09.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.599+0000 7f8c0a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.623 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.599+0000 7f8c0a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:09.623 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.600+0000 7f8c03fff640 10 monclient(hunting): get_auth_request con 0x7f8c0c1da150 auth_method 0 2023-10-30T20:24:09.623 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.600+0000 7f8c03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:09.623 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.600+0000 7f8c03fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.623 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.600+0000 7f8c03fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:09.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.600+0000 7f8c0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:09.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.600+0000 7f8c0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:09.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.600+0000 7f8c0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:09.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.600+0000 7f8c03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:09.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.600+0000 7f8c03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:09.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.600+0000 7f8c03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:09.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.601+0000 7f8c0ad9d640 1 --2- 172.21.15.69:0/3851739655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0c145770 0x7f8c0c1d3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.601+0000 7f8c0ad9d640 10 monclient(hunting): get_auth_request con 0x7f8c0c145770 auth_method 0 2023-10-30T20:24:09.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.601+0000 7f8c0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:09.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.601+0000 7f8c0ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.601+0000 7f8c0ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:09.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.602+0000 7f8c0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:09.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.602+0000 7f8c0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:09.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.602+0000 7f8c0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:09.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.602+0000 7f8c0a59c640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2023-10-30T20:24:09.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.602+0000 7f8c0a59c640 1 -- 172.21.15.69:0/3851739655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0c1da150 msgr2=0x7f8c0c1432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.602+0000 7f8c0a59c640 1 --2- 172.21.15.69:0/3851739655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0c1da150 0x7f8c0c1432d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.602+0000 7f8c0a59c640 1 -- 172.21.15.69:0/3851739655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0c145770 msgr2=0x7f8c0c1d3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.602+0000 7f8c0a59c640 1 --2- 172.21.15.69:0/3851739655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0c145770 0x7f8c0c1d3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.602+0000 7f8c0a59c640 10 monclient: _finish_hunting 0 2023-10-30T20:24:09.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.602+0000 7f8c0a59c640 1 monclient: found mon.noname-b 2023-10-30T20:24:09.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.603+0000 7f8c0a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:09.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.603+0000 7f8c0a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:24:09.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.603+0000 7f8c0a59c640 1 -- 172.21.15.69:0/3851739655 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c145250 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.628 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T20:24:09.629 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T20:24:09.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.605+0000 7f8c0a59c640 1 --2- 172.21.15.69:0/3851739655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c0c0a8e40 0x7f8c0c0ad930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc034040 tx=0x7f8bfc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:09.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.605+0000 7f8c02ffd640 10 client.?.objecter ms_handle_connect 0x7f8c0c0a8e40 2023-10-30T20:24:09.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.605+0000 7f8c02ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:09.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c02ffd640 1 -- 172.21.15.69:0/3851739655 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc032810 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c02ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:09.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c02ffd640 10 monclient: dump: 2023-10-30T20:24:09.632 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.632 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.632 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.632 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.633 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.633 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.633 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.633 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.633 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.634 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c02ffd640 10 monclient: _finish_auth 0 2023-10-30T20:24:09.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c02ffd640 10 monclient: _check_auth_tickets 2023-10-30T20:24:09.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:09.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c108af640 5 monclient: authenticate success, global_id 4348 2023-10-30T20:24:09.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c02ffd640 1 -- 172.21.15.69:0/3851739655 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bfc03f030 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c02ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:09.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c02ffd640 1 -- 172.21.15.69:0/3851739655 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc032be0 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c108af640 10 monclient: _renew_subs 2023-10-30T20:24:09.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c108af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:09.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c108af640 1 -- 172.21.15.69:0/3851739655 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0c0aca80 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c0959a640 4 set_mon_vals no callback set 2023-10-30T20:24:09.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:09.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c108af640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:09.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c02ffd640 10 monclient: dump: 2023-10-30T20:24:09.637 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.637 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.637 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.637 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.637 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.637 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.638 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.638 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.638 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.638 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c108af640 10 monclient: _renew_subs 2023-10-30T20:24:09.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c108af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:09.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.609+0000 7f8c108af640 1 -- 172.21.15.69:0/3851739655 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0c145bf0 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.610+0000 7f8c108af640 1 librados: init done 2023-10-30T20:24:09.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.611+0000 7f8c108af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:09.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.611+0000 7f8c108af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:09.640 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.611+0000 7f8c108af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:09.640 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.611+0000 7f8c108af640 1 -- 172.21.15.69:0/3851739655 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0c0a9220 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.684+0000 7ff75b7b0640 10 monclient: get_monmap_and_config 2023-10-30T20:24:09.687 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.684+0000 7ff75b7b0640 10 monclient: build_initial_monmap 2023-10-30T20:24:09.687 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.684+0000 7ff75b7b0640 10 monclient: monmap: 2023-10-30T20:24:09.687 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:09.687 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.687 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:09.686222+0000 2023-10-30T20:24:09.688 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:09.686222+0000 2023-10-30T20:24:09.688 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:09.688 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.688 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:09.688 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:09.688 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:09.689 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:09.689 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:09.689 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.689 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T20:24:09.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.684+0000 7ff75b7b0640 1 Processor -- start 2023-10-30T20:24:09.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 1 -- start start 2023-10-30T20:24:09.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 10 monclient: init 2023-10-30T20:24:09.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:09.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2023-10-30T20:24:09.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff7540a8e40 addr v1:172.21.15.69:6789/0 2023-10-30T20:24:09.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff7540b13c0 0x7ff754141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 10 monclient(hunting): picked mon.noname-c con 0x7ff7540b13c0 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:09.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7541423f0 0x7ff75414c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 10 monclient(hunting): picked mon.noname-b con 0x7ff7541423f0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:09.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7540b1b00 con 0x7ff7540a8e40 2023-10-30T20:24:09.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7540679a0 con 0x7ff7540b13c0 2023-10-30T20:24:09.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7540b1060 con 0x7ff7541423f0 2023-10-30T20:24:09.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:09.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.685+0000 7ff75b7b0640 10 monclient(hunting): authenticate will time out at 1572.975517s 2023-10-30T20:24:09.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.686+0000 7ff75954c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff7540a8e40 0x7ff75413efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36854/0 (socket says 172.21.15.69:36854) 2023-10-30T20:24:09.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.686+0000 7ff75954c640 1 -- 172.21.15.69:0/2942036321 learned_addr learned my addr 172.21.15.69:0/2942036321 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:24:09.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.686+0000 7ff759d4d640 1 --2- 172.21.15.69:0/2942036321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7541423f0 0x7ff75414c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.686+0000 7ff759d4d640 10 monclient(hunting): get_auth_request con 0x7ff7541423f0 auth_method 0 2023-10-30T20:24:09.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.686+0000 7ff759d4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:09.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.686+0000 7ff759d4d640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.686+0000 7ff759d4d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.687+0000 7ff759d4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.687+0000 7ff759d4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.687+0000 7ff759d4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.689+0000 7ff759d4d640 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2023-10-30T20:24:09.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.689+0000 7ff759d4d640 1 -- 172.21.15.69:0/2942036321 >> v2:172.21.15.183:3300/0 conn(0x7ff7540b13c0 msgr2=0x7ff754141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:09.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.689+0000 7ff759d4d640 1 --2- 172.21.15.69:0/2942036321 >> v2:172.21.15.183:3300/0 conn(0x7ff7540b13c0 0x7ff754141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.689+0000 7ff759d4d640 1 -- 172.21.15.69:0/2942036321 >> v1:172.21.15.69:6789/0 conn(0x7ff7540a8e40 legacy=0x7ff75413efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.689+0000 7ff759d4d640 10 monclient: _finish_hunting 0 2023-10-30T20:24:09.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.689+0000 7ff759d4d640 1 monclient: found mon.noname-b 2023-10-30T20:24:09.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.689+0000 7ff759d4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:09.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.689+0000 7ff759d4d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:24:09.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.689+0000 7ff759d4d640 1 -- 172.21.15.69:0/2942036321 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7540a8920 con 0x7ff7541423f0 2023-10-30T20:24:09.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.691+0000 7ff759d4d640 1 --2- 172.21.15.69:0/2942036321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7541423f0 0x7ff75414c320 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009d90 tx=0x7ff74c02bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cabd81e19442152 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:09.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff74bfff640 1 -- 172.21.15.69:0/2942036321 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c02cb30 con 0x7ff7541423f0 2023-10-30T20:24:09.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff74bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff74bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:09.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff74bfff640 10 monclient: dump: 2023-10-30T20:24:09.702 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.703 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.703 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.703 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.703 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.703 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.704 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.704 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.704 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.704 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff74bfff640 10 monclient: _finish_auth 0 2023-10-30T20:24:09.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff74bfff640 10 monclient: _check_auth_tickets 2023-10-30T20:24:09.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff74bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:09.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff74bfff640 1 -- 172.21.15.69:0/2942036321 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74c02cca0 con 0x7ff7541423f0 2023-10-30T20:24:09.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff74bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:09.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff74bfff640 1 -- 172.21.15.69:0/2942036321 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c034660 con 0x7ff7541423f0 2023-10-30T20:24:09.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff75b7b0640 5 monclient: authenticate success, global_id 4307 2023-10-30T20:24:09.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff75b7b0640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:09.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff75b7b0640 4 set_mon_vals no callback set 2023-10-30T20:24:09.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff74bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff74bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:09.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.696+0000 7ff74bfff640 10 monclient: dump: 2023-10-30T20:24:09.707 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.707 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.707 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.707 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.707 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.707 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.708 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.708 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.708 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.708 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.697+0000 7ff75b7b0640 10 monclient: shutdown 2023-10-30T20:24:09.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.697+0000 7ff75b7b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:09.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.697+0000 7ff75b7b0640 1 -- 172.21.15.69:0/2942036321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7541423f0 msgr2=0x7ff75414c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.697+0000 7ff75b7b0640 1 --2- 172.21.15.69:0/2942036321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7541423f0 0x7ff75414c320 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009d90 tx=0x7ff74c02bd80 comp rx=0 tx=0).stop 2023-10-30T20:24:09.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.697+0000 7ff75b7b0640 1 -- 172.21.15.69:0/2942036321 shutdown_connections 2023-10-30T20:24:09.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.697+0000 7ff75b7b0640 1 --2- 172.21.15.69:0/2942036321 >> v2:172.21.15.183:3300/0 conn(0x7ff7540b13c0 0x7ff754141eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.697+0000 7ff75b7b0640 1 --2- 172.21.15.69:0/2942036321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7541423f0 0x7ff75414c320 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.697+0000 7ff75b7b0640 1 -- 172.21.15.69:0/2942036321 >> 172.21.15.69:0/2942036321 conn(0x7ff7540b7180 msgr2=0x7ff7540b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:09.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.697+0000 7ff75b7b0640 1 -- 172.21.15.69:0/2942036321 shutdown_connections 2023-10-30T20:24:09.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.697+0000 7ff75b7b0640 1 -- 172.21.15.69:0/2942036321 wait complete. 2023-10-30T20:24:09.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.698+0000 7ff75b7b0640 10 monclient: build_initial_monmap 2023-10-30T20:24:09.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.698+0000 7ff75b7b0640 10 monclient: monmap: 2023-10-30T20:24:09.711 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:09.711 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:09.711 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:09.711 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:09.711 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:09.712 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:09.712 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:09.712 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:09.712 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:09.712 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.699+0000 7ff75b7b0640 1 librados: starting msgr at 2023-10-30T20:24:09.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.699+0000 7ff75b7b0640 1 librados: starting objecter 2023-10-30T20:24:09.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.699+0000 7ff75b7b0640 1 Processor -- start 2023-10-30T20:24:09.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.699+0000 7ff75b7b0640 1 -- start start 2023-10-30T20:24:09.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.699+0000 7ff75b7b0640 1 librados: setting wanted keys 2023-10-30T20:24:09.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.699+0000 7ff75b7b0640 1 librados: calling monclient init 2023-10-30T20:24:09.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.699+0000 7ff75b7b0640 10 monclient: init 2023-10-30T20:24:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T20:24:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7540a8e40 0x7ff7541dea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 10 monclient(hunting): picked mon.noname-b con 0x7ff7540a8e40 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:09.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7541423f0 0x7ff7541defd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 10 monclient(hunting): picked mon.noname-a con 0x7ff7541423f0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:09.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7541df510 0x7ff7541db320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 10 monclient(hunting): picked mon.noname-c con 0x7ff7541df510 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:09.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7540a83d0 con 0x7ff7541423f0 2023-10-30T20:24:09.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff754145d90 con 0x7ff7540a8e40 2023-10-30T20:24:09.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7541446c0 con 0x7ff7541df510 2023-10-30T20:24:09.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:09.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.700+0000 7ff75b7b0640 10 monclient(hunting): authenticate will time out at 1572.990400s 2023-10-30T20:24:09.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.701+0000 7ff758d4b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7541423f0 0x7ff7541defd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.701+0000 7ff759d4d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7541df510 0x7ff7541db320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.701+0000 7ff758d4b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7541423f0 0x7ff7541defd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56658/0 (socket says 172.21.15.69:56658) 2023-10-30T20:24:09.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.701+0000 7ff758d4b640 1 -- 172.21.15.69:0/4234578419 learned_addr learned my addr 172.21.15.69:0/4234578419 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:09.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.701+0000 7ff758d4b640 10 monclient(hunting): get_auth_request con 0x7ff7541423f0 auth_method 0 2023-10-30T20:24:09.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.701+0000 7ff758d4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:09.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.701+0000 7ff758d4b640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.701+0000 7ff758d4b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:09.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.701+0000 7ff759d4d640 10 monclient(hunting): get_auth_request con 0x7ff7541df510 auth_method 0 2023-10-30T20:24:09.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.701+0000 7ff759d4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:09.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.701+0000 7ff759d4d640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.701+0000 7ff759d4d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:09.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.702+0000 7ff758d4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:09.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.702+0000 7ff758d4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:09.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.702+0000 7ff758d4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:09.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.702+0000 7ff759d4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:09.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.702+0000 7ff759d4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:09.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.702+0000 7ff759d4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:09.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.705+0000 7ff758d4b640 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2023-10-30T20:24:09.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.705+0000 7ff758d4b640 1 -- 172.21.15.69:0/4234578419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7541df510 msgr2=0x7ff7541db320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.705+0000 7ff758d4b640 1 --2- 172.21.15.69:0/4234578419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7541df510 0x7ff7541db320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.705+0000 7ff758d4b640 1 -- 172.21.15.69:0/4234578419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7540a8e40 msgr2=0x7ff7541dea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.705+0000 7ff758d4b640 1 --2- 172.21.15.69:0/4234578419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7540a8e40 0x7ff7541dea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.705+0000 7ff758d4b640 10 monclient: _finish_hunting 0 2023-10-30T20:24:09.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.705+0000 7ff758d4b640 1 monclient: found mon.noname-a 2023-10-30T20:24:09.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.705+0000 7ff758d4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:09.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.705+0000 7ff758d4b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:09.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.705+0000 7ff758d4b640 1 -- 172.21.15.69:0/4234578419 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7540aacb0 con 0x7ff7541423f0 2023-10-30T20:24:09.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.707+0000 7ff758d4b640 1 --2- 172.21.15.69:0/4234578419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7541423f0 0x7ff7541defd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff74400b480 tx=0x7ff74400b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:09.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.707+0000 7ff749ffb640 10 client.?.objecter ms_handle_connect 0x7ff7541423f0 2023-10-30T20:24:09.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.707+0000 7ff749ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:09.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.710+0000 7ff749ffb640 1 -- 172.21.15.69:0/4234578419 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff744010870 con 0x7ff7541423f0 2023-10-30T20:24:09.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.710+0000 7ff749ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.710+0000 7ff749ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:09.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.710+0000 7ff749ffb640 10 monclient: dump: 2023-10-30T20:24:09.726 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.727 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.727 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.727 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.727 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.727 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.728 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.728 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.728 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.728 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.710+0000 7ff749ffb640 10 monclient: _finish_auth 0 2023-10-30T20:24:09.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.710+0000 7ff749ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:24:09.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.710+0000 7ff749ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:09.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.710+0000 7ff75b7b0640 5 monclient: authenticate success, global_id 4338 2023-10-30T20:24:09.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.710+0000 7ff75b7b0640 10 monclient: _renew_subs 2023-10-30T20:24:09.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.710+0000 7ff75b7b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:09.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.710+0000 7ff75b7b0640 1 -- 172.21.15.69:0/4234578419 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7540aaf70 con 0x7ff7541423f0 2023-10-30T20:24:09.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.711+0000 7ff75b7b0640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:09.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.711+0000 7ff75b7b0640 10 monclient: _renew_subs 2023-10-30T20:24:09.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.711+0000 7ff75b7b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:09.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.711+0000 7ff75b7b0640 1 -- 172.21.15.69:0/4234578419 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7540ab450 con 0x7ff7541423f0 2023-10-30T20:24:09.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.711+0000 7ff75b7b0640 1 librados: init done 2023-10-30T20:24:09.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.712+0000 7ff749ffb640 1 -- 172.21.15.69:0/4234578419 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff744009280 con 0x7ff7541423f0 2023-10-30T20:24:09.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.712+0000 7ff749ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:09.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.712+0000 7ff749ffb640 1 -- 172.21.15.69:0/4234578419 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff744010be0 con 0x7ff7541423f0 2023-10-30T20:24:09.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.712+0000 7ff749ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.712+0000 7ff749ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:09.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.712+0000 7ff749ffb640 10 monclient: dump: 2023-10-30T20:24:09.732 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.732 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.732 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.732 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.732 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.732 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.733 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.733 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.733 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.733 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.712+0000 7ff74b7fe640 4 set_mon_vals no callback set 2023-10-30T20:24:09.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.712+0000 7ff75b7b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:09.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.713+0000 7ff75b7b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:09.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.713+0000 7ff75b7b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:09.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.713+0000 7ff75b7b0640 1 -- 172.21.15.69:0/4234578419 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7540a9220 con 0x7ff7541423f0 2023-10-30T20:24:09.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.744+0000 7fe464475640 10 monclient: get_monmap_and_config 2023-10-30T20:24:09.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.744+0000 7fe464475640 10 monclient: build_initial_monmap 2023-10-30T20:24:09.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.745+0000 7fe464475640 10 monclient: monmap: 2023-10-30T20:24:09.748 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:09.748 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.748 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:09.746974+0000 2023-10-30T20:24:09.748 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:09.746974+0000 2023-10-30T20:24:09.748 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:09.749 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.749 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:09.749 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:09.749 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:09.749 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:09.749 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:09.750 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.753 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.745+0000 7fe464475640 1 Processor -- start 2023-10-30T20:24:09.753 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.745+0000 7fe464475640 1 -- start start 2023-10-30T20:24:09.753 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.745+0000 7fe464475640 10 monclient: init 2023-10-30T20:24:09.753 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.746+0000 7fe464475640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:09.753 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.746+0000 7fe464475640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-10-30T20:24:09.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.746+0000 7fe464475640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe45c0a8e40 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:09.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.746+0000 7fe464475640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe45c0b13c0 0x7fe45c141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.746+0000 7fe464475640 10 monclient(hunting): picked mon.noname-a con 0x7fe45c0b13c0 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:09.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.746+0000 7fe464475640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe45c1423f0 0x7fe45c14c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.746+0000 7fe464475640 10 monclient(hunting): picked mon.noname-b con 0x7fe45c1423f0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:09.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.746+0000 7fe464475640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe45c0b1b00 con 0x7fe45c0a8e40 2023-10-30T20:24:09.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.746+0000 7fe464475640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.746+0000 7fe464475640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe45c0679a0 con 0x7fe45c0b13c0 2023-10-30T20:24:09.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.746+0000 7fe464475640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.746+0000 7fe464475640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c0b1060 con 0x7fe45c1423f0 2023-10-30T20:24:09.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.746+0000 7fe464475640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:09.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.746+0000 7fe464475640 10 monclient(hunting): authenticate will time out at 1573.036466s 2023-10-30T20:24:09.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.748+0000 7fe462a12640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe45c1423f0 0x7fe45c14c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.748+0000 7fe462a12640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe45c1423f0 0x7fe45c14c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:40604/0 (socket says 172.21.15.69:40604) 2023-10-30T20:24:09.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.748+0000 7fe462a12640 1 -- 172.21.15.69:0/2470261409 learned_addr learned my addr 172.21.15.69:0/2470261409 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:09.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.748+0000 7fe462a12640 10 monclient(hunting): get_auth_request con 0x7fe45c1423f0 auth_method 0 2023-10-30T20:24:09.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.748+0000 7fe462a12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:09.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.748+0000 7fe462a12640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.748+0000 7fe462a12640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:09.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.748+0000 7fe462a12640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:09.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.748+0000 7fe462a12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:09.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.748+0000 7fe462a12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:09.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.750+0000 7fe462a12640 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2023-10-30T20:24:09.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.750+0000 7fe462a12640 1 -- 172.21.15.69:0/2470261409 >> v2:172.21.15.69:3300/0 conn(0x7fe45c0b13c0 msgr2=0x7fe45c141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:09.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.750+0000 7fe462a12640 1 --2- 172.21.15.69:0/2470261409 >> v2:172.21.15.69:3300/0 conn(0x7fe45c0b13c0 0x7fe45c141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.750+0000 7fe462a12640 1 -- 172.21.15.69:0/2470261409 >> v1:172.21.15.183:6789/0 conn(0x7fe45c0a8e40 legacy=0x7fe45c13efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:09.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.750+0000 7fe462a12640 10 monclient: _finish_hunting 0 2023-10-30T20:24:09.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.750+0000 7fe462a12640 1 monclient: found mon.noname-b 2023-10-30T20:24:09.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.750+0000 7fe462a12640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:09.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.750+0000 7fe462a12640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:24:09.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.750+0000 7fe462a12640 1 -- 172.21.15.69:0/2470261409 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c0a8920 con 0x7fe45c1423f0 2023-10-30T20:24:09.762 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.754+0000 7fe462a12640 1 --2- 172.21.15.69:0/2470261409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe45c1423f0 0x7fe45c14c320 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe454009e50 tx=0x7fe45402bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9b9fdff5cbff60e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:09.762 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.755+0000 7fe46120f640 1 -- 172.21.15.69:0/2470261409 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45402c950 con 0x7fe45c1423f0 2023-10-30T20:24:09.762 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.756+0000 7fe46120f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.763 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.756+0000 7fe46120f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:09.763 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.756+0000 7fe46120f640 10 monclient: dump: 2023-10-30T20:24:09.763 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.763 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.763 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.764 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.764 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.764 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.764 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.764 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.765 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.765 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.765 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.756+0000 7fe46120f640 10 monclient: _finish_auth 0 2023-10-30T20:24:09.765 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.756+0000 7fe46120f640 10 monclient: _check_auth_tickets 2023-10-30T20:24:09.765 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.756+0000 7fe46120f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:09.765 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.756+0000 7fe464475640 5 monclient: authenticate success, global_id 4313 2023-10-30T20:24:09.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.756+0000 7fe464475640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:09.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.758+0000 7fe46120f640 1 -- 172.21.15.69:0/2470261409 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45402cac0 con 0x7fe45c1423f0 2023-10-30T20:24:09.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.758+0000 7fe46120f640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:09.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.758+0000 7fe464475640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:09.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.758+0000 7fe464475640 4 set_mon_vals no callback set 2023-10-30T20:24:09.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.758+0000 7fe46120f640 1 -- 172.21.15.69:0/2470261409 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45402c950 con 0x7fe45c1423f0 2023-10-30T20:24:09.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.758+0000 7fe46120f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.758+0000 7fe46120f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:09.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.758+0000 7fe46120f640 10 monclient: dump: 2023-10-30T20:24:09.767 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.768 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.768 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.768 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.768 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.768 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.769 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.769 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.769 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.769 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.759+0000 7fe464475640 10 monclient: shutdown 2023-10-30T20:24:09.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.759+0000 7fe464475640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.759+0000 7fe464475640 1 -- 172.21.15.69:0/2470261409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe45c1423f0 msgr2=0x7fe45c14c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.759+0000 7fe464475640 1 --2- 172.21.15.69:0/2470261409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe45c1423f0 0x7fe45c14c320 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe454009e50 tx=0x7fe45402bdb0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.759+0000 7fe464475640 1 -- 172.21.15.69:0/2470261409 shutdown_connections 2023-10-30T20:24:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.759+0000 7fe464475640 1 --2- 172.21.15.69:0/2470261409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe45c1423f0 0x7fe45c14c320 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.759+0000 7fe464475640 1 --2- 172.21.15.69:0/2470261409 >> v2:172.21.15.69:3300/0 conn(0x7fe45c0b13c0 0x7fe45c141eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.771 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.759+0000 7fe464475640 1 -- 172.21.15.69:0/2470261409 >> 172.21.15.69:0/2470261409 conn(0x7fe45c0b7180 msgr2=0x7fe45c0b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:09.771 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.759+0000 7fe464475640 1 -- 172.21.15.69:0/2470261409 shutdown_connections 2023-10-30T20:24:09.771 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.759+0000 7fe464475640 1 -- 172.21.15.69:0/2470261409 wait complete. 2023-10-30T20:24:09.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.760+0000 7fe464475640 10 monclient: build_initial_monmap 2023-10-30T20:24:09.774 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.760+0000 7fe464475640 10 monclient: monmap: 2023-10-30T20:24:09.774 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:09.774 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:09.774 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:09.774 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:09.775 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:09.775 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:09.775 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:09.775 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:09.775 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:09.776 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.760+0000 7fe464475640 1 librados: starting msgr at 2023-10-30T20:24:09.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.760+0000 7fe464475640 1 librados: starting objecter 2023-10-30T20:24:09.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 1 Processor -- start 2023-10-30T20:24:09.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 1 -- start start 2023-10-30T20:24:09.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 1 librados: setting wanted keys 2023-10-30T20:24:09.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 1 librados: calling monclient init 2023-10-30T20:24:09.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 10 monclient: init 2023-10-30T20:24:09.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:09.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:24:09.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe45c0a8e40 0x7fe45c139b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 10 monclient(hunting): picked mon.noname-a con 0x7fe45c0a8e40 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:09.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe45c1423f0 0x7fe45c0ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 10 monclient(hunting): picked mon.noname-b con 0x7fe45c1423f0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:09.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe45c0ab5c0 0x7fe45c0ada50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 10 monclient(hunting): picked mon.noname-c con 0x7fe45c0ab5c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:09.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c0b3590 con 0x7fe45c0a8e40 2023-10-30T20:24:09.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c0adf90 con 0x7fe45c1423f0 2023-10-30T20:24:09.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c13f8f0 con 0x7fe45c0ab5c0 2023-10-30T20:24:09.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:09.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.761+0000 7fe464475640 10 monclient(hunting): authenticate will time out at 1573.051612s 2023-10-30T20:24:09.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.762+0000 7fe462a12640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe45c0ab5c0 0x7fe45c0ada50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.762+0000 7fe462a12640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe45c0ab5c0 0x7fe45c0ada50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:40618/0 (socket says 172.21.15.69:40618) 2023-10-30T20:24:09.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.762+0000 7fe462a12640 1 -- 172.21.15.69:0/590401767 learned_addr learned my addr 172.21.15.69:0/590401767 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:09.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.762+0000 7fe462a12640 10 monclient(hunting): get_auth_request con 0x7fe45c0ab5c0 auth_method 0 2023-10-30T20:24:09.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.762+0000 7fe462a12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:09.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.762+0000 7fe462a12640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.762+0000 7fe462a12640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:09.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.763+0000 7fe462a12640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:09.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.763+0000 7fe462a12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:09.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.763+0000 7fe462a12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:09.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.764+0000 7f4a49c83640 10 monclient: get_monmap_and_config 2023-10-30T20:24:09.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.764+0000 7f4a49c83640 10 monclient: build_initial_monmap 2023-10-30T20:24:09.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.764+0000 7f4a49c83640 10 monclient: monmap: 2023-10-30T20:24:09.784 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:09.784 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.784 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:09.766539+0000 2023-10-30T20:24:09.784 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:09.766539+0000 2023-10-30T20:24:09.784 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:09.784 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.785 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:09.785 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:09.785 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:09.785 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:09.785 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:09.786 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.765+0000 7f4a49c83640 1 Processor -- start 2023-10-30T20:24:09.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.765+0000 7f4a49c83640 1 -- start start 2023-10-30T20:24:09.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.765+0000 7f4a49c83640 10 monclient: init 2023-10-30T20:24:09.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7f4a49c83640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:09.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7f4a49c83640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-10-30T20:24:09.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7f4a49c83640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4a440a8e40 0x7f4a4413efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7f4a49c83640 10 monclient(hunting): picked mon.noname-a con 0x7f4a440a8e40 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:09.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7f4a49c83640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a440b13c0 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:09.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7f4a49c83640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a4413f500 addr v1:172.21.15.69:6789/0 2023-10-30T20:24:09.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7f4a49c83640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a440b1b00 con 0x7f4a4413f500 2023-10-30T20:24:09.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7f4a49c83640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a440a8920 con 0x7f4a440b13c0 2023-10-30T20:24:09.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7f4a49c83640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7f4a49c83640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a440679a0 con 0x7f4a440a8e40 2023-10-30T20:24:09.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7f4a49c83640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:09.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7f4a49c83640 10 monclient(hunting): authenticate will time out at 1573.056222s 2023-10-30T20:24:09.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7fe462a12640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2023-10-30T20:24:09.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7fe462a12640 1 -- 172.21.15.69:0/590401767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe45c1423f0 msgr2=0x7fe45c0ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7fe462a12640 1 --2- 172.21.15.69:0/590401767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe45c1423f0 0x7fe45c0ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7fe462a12640 1 -- 172.21.15.69:0/590401767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe45c0a8e40 msgr2=0x7fe45c139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7fe462a12640 1 --2- 172.21.15.69:0/590401767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe45c0a8e40 0x7fe45c139b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7fe462a12640 10 monclient: _finish_hunting 0 2023-10-30T20:24:09.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7fe462a12640 1 monclient: found mon.noname-c 2023-10-30T20:24:09.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7fe462a12640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:09.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7fe462a12640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:24:09.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.766+0000 7fe462a12640 1 -- 172.21.15.69:0/590401767 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c140c90 con 0x7fe45c0ab5c0 2023-10-30T20:24:09.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.769+0000 7fe462a12640 1 --2- 172.21.15.69:0/590401767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe45c0ab5c0 0x7fe45c0ada50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe454009f70 tx=0x7fe454004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:09.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.769+0000 7fe452ffd640 10 client.?.objecter ms_handle_connect 0x7fe45c0ab5c0 2023-10-30T20:24:09.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.769+0000 7fe452ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:09.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.773+0000 7fe452ffd640 1 -- 172.21.15.69:0/590401767 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454004040 con 0x7fe45c0ab5c0 2023-10-30T20:24:09.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.773+0000 7fe452ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.773+0000 7fe452ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:09.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.773+0000 7fe452ffd640 10 monclient: dump: 2023-10-30T20:24:09.793 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.793 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.793 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.794 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.794 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.794 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.794 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.794 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.794 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.795 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.773+0000 7fe452ffd640 10 monclient: _finish_auth 0 2023-10-30T20:24:09.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.773+0000 7fe452ffd640 10 monclient: _check_auth_tickets 2023-10-30T20:24:09.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.773+0000 7fe452ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:09.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.773+0000 7fe452ffd640 1 -- 172.21.15.69:0/590401767 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4540041b0 con 0x7fe45c0ab5c0 2023-10-30T20:24:09.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.773+0000 7fe464475640 5 monclient: authenticate success, global_id 4316 2023-10-30T20:24:09.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.773+0000 7fe464475640 10 monclient: _renew_subs 2023-10-30T20:24:09.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.773+0000 7fe464475640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:09.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.773+0000 7fe464475640 1 -- 172.21.15.69:0/590401767 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe45c0ab170 con 0x7fe45c0ab5c0 2023-10-30T20:24:09.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.774+0000 7fe452ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:09.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.774+0000 7fe452ffd640 1 -- 172.21.15.69:0/590401767 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454004370 con 0x7fe45c0ab5c0 2023-10-30T20:24:09.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.774+0000 7fe452ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.774+0000 7fe452ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:09.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.774+0000 7fe452ffd640 10 monclient: dump: 2023-10-30T20:24:09.797 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.797 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.797 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.798 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.798 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.798 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.798 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.798 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.799 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.799 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.799 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.774+0000 7fe460a0e640 4 set_mon_vals no callback set 2023-10-30T20:24:09.799 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.774+0000 7fe464475640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:09.799 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.774+0000 7fe464475640 10 monclient: _renew_subs 2023-10-30T20:24:09.799 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.774+0000 7fe464475640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:09.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.774+0000 7fe464475640 1 -- 172.21.15.69:0/590401767 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe45c13ffc0 con 0x7fe45c0ab5c0 2023-10-30T20:24:09.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.774+0000 7fe464475640 1 librados: init done 2023-10-30T20:24:09.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.774+0000 7fe450ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:09.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.775+0000 7fe450ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:09.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.775+0000 7fe450ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:09.801 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.775+0000 7fe450ff9640 1 -- 172.21.15.69:0/590401767 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe438005040 con 0x7fe45c0ab5c0 2023-10-30T20:24:09.803 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.801+0000 7f47a2bbb640 10 monclient: get_monmap_and_config 2023-10-30T20:24:09.804 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.801+0000 7f47a2bbb640 10 monclient: build_initial_monmap 2023-10-30T20:24:09.804 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.801+0000 7f47a2bbb640 10 monclient: monmap: 2023-10-30T20:24:09.804 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:09.804 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.804 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:09.803738+0000 2023-10-30T20:24:09.805 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:09.803738+0000 2023-10-30T20:24:09.805 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:09.805 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.805 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:09.805 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:09.806 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:09.806 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:09.806 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:09.806 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.802+0000 7f47a2bbb640 1 Processor -- start 2023-10-30T20:24:09.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.802+0000 7f47a2bbb640 1 -- start start 2023-10-30T20:24:09.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.802+0000 7f47a2bbb640 10 monclient: init 2023-10-30T20:24:09.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.803+0000 7f47a2bbb640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:09.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.803+0000 7f47a2bbb640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-10-30T20:24:09.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.803+0000 7f47a2bbb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f479c0a8e40 0x7f479c13efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.803+0000 7f47a2bbb640 10 monclient(hunting): picked mon.noname-c con 0x7f479c0a8e40 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:09.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.803+0000 7f47a2bbb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f479c0b13c0 0x7f479c141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.803+0000 7f47a2bbb640 10 monclient(hunting): picked mon.noname-a con 0x7f479c0b13c0 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:09.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.803+0000 7f47a2bbb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f479c13f500 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:09.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.803+0000 7f47a2bbb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f479c0b1b00 con 0x7f479c13f500 2023-10-30T20:24:09.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.803+0000 7f47a2bbb640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.803+0000 7f47a2bbb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479c0679a0 con 0x7f479c0b13c0 2023-10-30T20:24:09.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.803+0000 7f47a2bbb640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.803+0000 7f47a2bbb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479c0b1060 con 0x7f479c0a8e40 2023-10-30T20:24:09.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.803+0000 7f47a2bbb640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:09.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.803+0000 7f47a2bbb640 10 monclient(hunting): authenticate will time out at 1573.092934s 2023-10-30T20:24:09.823 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T20:24:09.830 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T20:24:09.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.850+0000 7f8c02ffd640 1 -- 172.21.15.69:0/3851739655 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f8bfc032d50 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.851+0000 7f8c02ffd640 1 --2- 172.21.15.69:0/3851739655 >> v2:172.21.15.69:6800/1876307353 conn(0x7f8bec06c7c0 0x7f8bec06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.851+0000 7f8c03fff640 1 --2- 172.21.15.69:0/3851739655 >> v2:172.21.15.69:6800/1876307353 conn(0x7f8bec06c7c0 0x7f8bec06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.851+0000 7f8c03fff640 10 monclient: get_auth_request con 0x7f8bec06c7c0 auth_method 0 2023-10-30T20:24:09.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.851+0000 7f8c02ffd640 10 client.4348.objecter ms_handle_connect 0x7f8bec06c7c0 2023-10-30T20:24:09.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.851+0000 7f8c03fff640 1 --2- 172.21.15.69:0/3851739655 >> v2:172.21.15.69:6800/1876307353 conn(0x7f8bec06c7c0 0x7f8bec06ec80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4001380 tx=0x7f8bf4007b90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:09.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.852+0000 7f8c02ffd640 1 -- 172.21.15.69:0/3851739655 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f8bfc053080 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.852+0000 7f8c02ffd640 10 client.4348.objecter ms_dispatch 0x7f8c0c000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:09.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.852+0000 7f8c02ffd640 3 client.4348.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:09.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.852+0000 7f8c02ffd640 3 client.4348.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:09.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.852+0000 7f8c02ffd640 20 client.4348.objecter dump_active .. 0 homeless 2023-10-30T20:24:09.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.862+0000 7f8c02ffd640 1 -- 172.21.15.69:0/3851739655 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8bfc037070 con 0x7f8c0c0a8e40 2023-10-30T20:24:09.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.862+0000 7f8c02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:09.865 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.862+0000 7f8c02ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:09.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.867+0000 7f47a0957640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f479c0a8e40 0x7f479c13efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.867+0000 7f47a0957640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f479c0a8e40 0x7f479c13efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:53120/0 (socket says 172.21.15.69:53120) 2023-10-30T20:24:09.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.867+0000 7f47a0957640 1 -- 172.21.15.69:0/3451376854 learned_addr learned my addr 172.21.15.69:0/3451376854 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:09.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.867+0000 7f47a0957640 10 monclient(hunting): get_auth_request con 0x7f479c0a8e40 auth_method 0 2023-10-30T20:24:09.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.867+0000 7f47a0957640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:09.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.867+0000 7f47a0957640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.867+0000 7f47a0957640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:09.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.868+0000 7f47a0957640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:09.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.868+0000 7f47a0957640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:09.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.868+0000 7f47a0957640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:09.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.869+0000 7f4a42ffd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f4a440b13c0 0x7f4a44141eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.69:51854/0 (socket says 172.21.15.69:51854) 2023-10-30T20:24:09.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.869+0000 7f4a42ffd640 1 -- 172.21.15.69:0/4124229765 learned_addr learned my addr 172.21.15.69:0/4124229765 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:24:09.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.870+0000 7f47a0957640 10 monclient(hunting): handle_auth_done global_id 4351 payload 274 2023-10-30T20:24:09.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.870+0000 7f47a0957640 1 -- 172.21.15.69:0/3451376854 >> v2:172.21.15.69:3300/0 conn(0x7f479c0b13c0 msgr2=0x7f479c141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:09.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.870+0000 7f47a0957640 1 --2- 172.21.15.69:0/3451376854 >> v2:172.21.15.69:3300/0 conn(0x7f479c0b13c0 0x7f479c141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.870+0000 7f47a0957640 1 -- 172.21.15.69:0/3451376854 >> v1:172.21.15.183:6789/0 conn(0x7f479c13f500 legacy=0x7f479c1423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:09.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.870+0000 7f47a0957640 10 monclient: _finish_hunting 0 2023-10-30T20:24:09.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.870+0000 7f47a0957640 1 monclient: found mon.noname-c 2023-10-30T20:24:09.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.870+0000 7f47a0957640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:09.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.870+0000 7f47a0957640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:24:09.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.870+0000 7f47a0957640 1 -- 172.21.15.69:0/3451376854 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c0a8920 con 0x7f479c0a8e40 2023-10-30T20:24:09.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.873+0000 7f47a0957640 1 --2- 172.21.15.69:0/3451376854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f479c0a8e40 0x7f479c13efc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f479c0b8030 tx=0x7f479002b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4714c318abd73e17 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:09.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.877+0000 7f4a427fc640 1 -- 172.21.15.69:0/4124229765 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401638406 0 0) 0x7f4a38001110 con 0x7f4a440b13c0 2023-10-30T20:24:09.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.877+0000 7f4a427fc640 10 monclient(hunting): _negotiate 2023-10-30T20:24:09.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.877+0000 7f4a427fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.877+0000 7f4a427fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:09.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.877+0000 7f4a427fc640 10 monclient(hunting): my global_id is 4354 2023-10-30T20:24:09.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.877+0000 7f4a427fc640 1 -- 172.21.15.69:0/4124229765 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a2c003520 con 0x7f4a440b13c0 2023-10-30T20:24:09.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f479b7fe640 1 -- 172.21.15.69:0/3451376854 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790031750 con 0x7f479c0a8e40 2023-10-30T20:24:09.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f479b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f479b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:09.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f479b7fe640 10 monclient: dump: 2023-10-30T20:24:09.885 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.886 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.886 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.886 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.886 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.886 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.887 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.887 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.887 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.887 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f479b7fe640 10 monclient: _finish_auth 0 2023-10-30T20:24:09.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f479b7fe640 10 monclient: _check_auth_tickets 2023-10-30T20:24:09.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f479b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:09.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f479b7fe640 1 -- 172.21.15.69:0/3451376854 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47900318c0 con 0x7f479c0a8e40 2023-10-30T20:24:09.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f47a2bbb640 5 monclient: authenticate success, global_id 4351 2023-10-30T20:24:09.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f47a2bbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:09.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f479b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:09.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f479b7fe640 1 -- 172.21.15.69:0/3451376854 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790031c60 con 0x7f479c0a8e40 2023-10-30T20:24:09.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f47a2bbb640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:09.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f47a2bbb640 4 set_mon_vals no callback set 2023-10-30T20:24:09.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f479b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f479b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:09.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.882+0000 7f479b7fe640 10 monclient: dump: 2023-10-30T20:24:09.890 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.890 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.890 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.891 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.891 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.891 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.891 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.891 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.892 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.892 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 -- 172.21.15.69:0/4124229765 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 130281126 0 0) 0x7f4a38001510 con 0x7f4a440b13c0 2023-10-30T20:24:09.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 -- 172.21.15.69:0/4124229765 >> v2:172.21.15.69:3300/0 conn(0x7f4a440a8e40 msgr2=0x7f4a4413efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:09.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 --2- 172.21.15.69:0/4124229765 >> v2:172.21.15.69:3300/0 conn(0x7f4a440a8e40 0x7f4a4413efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 -- 172.21.15.69:0/4124229765 >> v1:172.21.15.69:6789/0 conn(0x7f4a4413f500 legacy=0x7f4a4414c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:09.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient: _finish_hunting 0 2023-10-30T20:24:09.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 monclient: found mon.noname-c-legacy 2023-10-30T20:24:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.183:6789/0 2023-10-30T20:24:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 -- 172.21.15.69:0/4124229765 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a440a97b0 con 0x7f4a440b13c0 2023-10-30T20:24:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient: _finish_auth 0 2023-10-30T20:24:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient: _check_auth_tickets 2023-10-30T20:24:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:09.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 -- 172.21.15.69:0/4124229765 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 396370546 0 0) 0x7f4a380019b0 con 0x7f4a440b13c0 2023-10-30T20:24:09.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a49c83640 5 monclient: authenticate success, global_id 4354 2023-10-30T20:24:09.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f47a2bbb640 10 monclient: shutdown 2023-10-30T20:24:09.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f47a2bbb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:09.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f47a2bbb640 1 -- 172.21.15.69:0/3451376854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f479c0a8e40 msgr2=0x7f479c13efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f47a2bbb640 1 --2- 172.21.15.69:0/3451376854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f479c0a8e40 0x7f479c13efc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f479c0b8030 tx=0x7f479002b7f0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a49c83640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:09.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f47a2bbb640 1 -- 172.21.15.69:0/3451376854 shutdown_connections 2023-10-30T20:24:09.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f47a2bbb640 1 --2- 172.21.15.69:0/3451376854 >> v2:172.21.15.69:3300/0 conn(0x7f479c0b13c0 0x7f479c141eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f47a2bbb640 1 --2- 172.21.15.69:0/3451376854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f479c0a8e40 0x7f479c13efc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f47a2bbb640 1 -- 172.21.15.69:0/3451376854 >> 172.21.15.69:0/3451376854 conn(0x7f479c0b7180 msgr2=0x7f479c0b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:09.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T20:24:09.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient: dump: 2023-10-30T20:24:09.898 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.898 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.899 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.899 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.899 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.899 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.899 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.899 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.900 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] but i'm connected to v1:172.21.15.183:6789/0, reconnecting 2023-10-30T20:24:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 -- 172.21.15.69:0/4124229765 >> v1:172.21.15.183:6789/0 conn(0x7f4a440b13c0 legacy=0x7f4a44141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:24:09.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 --2- 172.21.15.69:0/4124229765 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c003970 0x7f4a2c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient(hunting): picked mon.b con 0x7f4a2c003970 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:09.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 --2- 172.21.15.69:0/4124229765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2c0047c0 0x7f4a2c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient(hunting): picked mon.c con 0x7f4a2c0047c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:09.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 --2- 172.21.15.69:0/4124229765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a2c005380 0x7f4a2c00b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient(hunting): picked mon.a con 0x7f4a2c005380 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:09.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 -- 172.21.15.69:0/4124229765 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440679a0 con 0x7f4a2c005380 2023-10-30T20:24:09.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 -- 172.21.15.69:0/4124229765 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c00c2c0 con 0x7f4a2c003970 2023-10-30T20:24:09.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 1 -- 172.21.15.69:0/4124229765 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c014ca0 con 0x7f4a2c0047c0 2023-10-30T20:24:09.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:09.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a427fc640 10 monclient(hunting): _finish_auth 0 2023-10-30T20:24:09.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f4a49c83640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:09.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f47a2bbb640 1 -- 172.21.15.69:0/3451376854 shutdown_connections 2023-10-30T20:24:09.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.883+0000 7f47a2bbb640 1 -- 172.21.15.69:0/3451376854 wait complete. 2023-10-30T20:24:09.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.884+0000 7f47a2bbb640 10 monclient: build_initial_monmap 2023-10-30T20:24:09.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.884+0000 7f47a2bbb640 10 monclient: monmap: 2023-10-30T20:24:09.905 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:09.905 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:09.905 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:09.905 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:09.905 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:09.906 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:09.906 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:09.906 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:09.906 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:09.907 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.885+0000 7f4a437fe640 1 --2- 172.21.15.69:0/4124229765 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c003970 0x7f4a2c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.885+0000 7f47a2bbb640 1 librados: starting msgr at 2023-10-30T20:24:09.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.885+0000 7f47a2bbb640 1 librados: starting objecter 2023-10-30T20:24:09.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.885+0000 7f47a2bbb640 1 Processor -- start 2023-10-30T20:24:09.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.885+0000 7f47a2bbb640 1 -- start start 2023-10-30T20:24:09.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.885+0000 7f47a2bbb640 1 librados: setting wanted keys 2023-10-30T20:24:09.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.885+0000 7f47a2bbb640 1 librados: calling monclient init 2023-10-30T20:24:09.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.885+0000 7f47a2bbb640 10 monclient: init 2023-10-30T20:24:09.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f4a437fe640 10 monclient(hunting): get_auth_request con 0x7f4a2c003970 auth_method 0 2023-10-30T20:24:09.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f4a437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:09.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f4a437fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f4a437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:24:09.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:09.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:24:09.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f479c0a8e40 0x7f479c0af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 10 monclient(hunting): picked mon.noname-b con 0x7f479c0a8e40 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:09.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f479c139670 0x7f479c1de180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 10 monclient(hunting): picked mon.noname-c con 0x7f479c139670 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:09.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f479c13c7d0 0x7f479c1de6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 10 monclient(hunting): picked mon.noname-a con 0x7f479c13c7d0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:09.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c0b07c0 con 0x7f479c13c7d0 2023-10-30T20:24:09.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c13a3b0 con 0x7f479c0a8e40 2023-10-30T20:24:09.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f479c13a580 con 0x7f479c139670 2023-10-30T20:24:09.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:09.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f47a2bbb640 10 monclient(hunting): authenticate will time out at 1573.176384s 2023-10-30T20:24:09.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f4a437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:09.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f4a437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:09.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.886+0000 7f4a437fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T20:24:09.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.888+0000 7f4a437fe640 10 monclient(hunting): handle_auth_done global_id 4354 payload 293 2023-10-30T20:24:09.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.888+0000 7f4a437fe640 1 -- 172.21.15.69:0/4124229765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2c0047c0 msgr2=0x7f4a2c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:09.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.888+0000 7f4a437fe640 1 --2- 172.21.15.69:0/4124229765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2c0047c0 0x7f4a2c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.888+0000 7f4a437fe640 1 -- 172.21.15.69:0/4124229765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a2c005380 msgr2=0x7f4a2c00b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:09.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.888+0000 7f4a437fe640 1 --2- 172.21.15.69:0/4124229765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a2c005380 0x7f4a2c00b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.888+0000 7f4a437fe640 10 monclient: _finish_hunting 0 2023-10-30T20:24:09.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.889+0000 7f4a437fe640 1 monclient: found mon.b 2023-10-30T20:24:09.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.889+0000 7f4a43fff640 1 -- 172.21.15.69:0/4124229765 reap_dead start 2023-10-30T20:24:09.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.889+0000 7f4a437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:09.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.889+0000 7f4a437fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:09.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.889+0000 7f4a437fe640 1 -- 172.21.15.69:0/4124229765 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a440a97b0 con 0x7f4a2c003970 2023-10-30T20:24:09.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.889+0000 7f4a437fe640 10 monclient: _finish_auth 0 2023-10-30T20:24:09.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.889+0000 7f4a437fe640 10 monclient: _check_auth_tickets 2023-10-30T20:24:09.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.889+0000 7f4a437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:09.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.889+0000 7f47a0957640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f479c0a8e40 0x7f479c0af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.890+0000 7f47a0957640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f479c0a8e40 0x7f479c0af740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:53138/0 (socket says 172.21.15.69:53138) 2023-10-30T20:24:09.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.890+0000 7f47a0957640 1 -- 172.21.15.69:0/4109089910 learned_addr learned my addr 172.21.15.69:0/4109089910 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:09.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.890+0000 7f47a0957640 10 monclient(hunting): get_auth_request con 0x7f479c0a8e40 auth_method 0 2023-10-30T20:24:09.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.890+0000 7f47a0957640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:09.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.890+0000 7f47a0957640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:09.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.890+0000 7f47a0957640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:09.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.891+0000 7f47a0957640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:09.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.891+0000 7f47a0957640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:09.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.891+0000 7f47a0957640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:09.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.892+0000 7f4a437fe640 1 --2- 172.21.15.69:0/4124229765 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c003970 0x7f4a2c003d70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4a34009860 tx=0x7f4a2c0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=266a7cd298a8770d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:09.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.894+0000 7f47a0957640 10 monclient(hunting): handle_auth_done global_id 4357 payload 824 2023-10-30T20:24:09.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.894+0000 7f47a0957640 1 -- 172.21.15.69:0/4109089910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f479c139670 msgr2=0x7f479c1de180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.894+0000 7f47a0957640 1 --2- 172.21.15.69:0/4109089910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f479c139670 0x7f479c1de180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.894+0000 7f47a0957640 1 -- 172.21.15.69:0/4109089910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f479c13c7d0 msgr2=0x7f479c1de6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.894+0000 7f47a0957640 1 --2- 172.21.15.69:0/4109089910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f479c13c7d0 0x7f479c1de6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.894+0000 7f47a0957640 10 monclient: _finish_hunting 0 2023-10-30T20:24:09.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.894+0000 7f47a0957640 1 monclient: found mon.noname-b 2023-10-30T20:24:09.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.894+0000 7f47a0957640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:09.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.894+0000 7f47a0957640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:24:09.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.894+0000 7f47a0957640 1 -- 172.21.15.69:0/4109089910 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c13c460 con 0x7f479c0a8e40 2023-10-30T20:24:09.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.899+0000 7f47a0957640 1 --2- 172.21.15.69:0/4109089910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f479c0a8e40 0x7f479c0af740 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f479002b830 tx=0x7f47900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:09.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.899+0000 7f47997fa640 10 client.?.objecter ms_handle_connect 0x7f479c0a8e40 2023-10-30T20:24:09.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.899+0000 7f47997fa640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:09.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.900+0000 7f4a427fc640 1 -- 172.21.15.69:0/4124229765 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3402e050 con 0x7f4a2c003970 2023-10-30T20:24:09.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.900+0000 7f4a427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.900+0000 7f4a427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:09.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.900+0000 7f4a427fc640 10 monclient: dump: 2023-10-30T20:24:09.927 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.928 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.928 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.928 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.928 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.928 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.929 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.900+0000 7f4a49c83640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:09.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.900+0000 7f4a427fc640 1 -- 172.21.15.69:0/4124229765 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a34032600 con 0x7f4a2c003970 2023-10-30T20:24:09.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.900+0000 7f4a427fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:09.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.900+0000 7f4a49c83640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:09.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.900+0000 7f4a49c83640 4 set_mon_vals no callback set 2023-10-30T20:24:09.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.901+0000 7f4a49c83640 10 monclient: shutdown 2023-10-30T20:24:09.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.901+0000 7f4a427fc640 1 -- 172.21.15.69:0/4124229765 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3402e050 con 0x7f4a2c003970 2023-10-30T20:24:09.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.901+0000 7f4a427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.901+0000 7f4a427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:09.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.901+0000 7f4a427fc640 10 monclient: dump: 2023-10-30T20:24:09.931 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.931 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.932 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.932 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.932 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.932 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.932 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.933 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.933 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.933 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.901+0000 7f4a49c83640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:09.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.901+0000 7f4a49c83640 1 -- 172.21.15.69:0/4124229765 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c003970 msgr2=0x7f4a2c003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:09.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.901+0000 7f4a49c83640 1 --2- 172.21.15.69:0/4124229765 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c003970 0x7f4a2c003d70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4a34009860 tx=0x7f4a2c0057f0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.902+0000 7f4a49c83640 1 -- 172.21.15.69:0/4124229765 shutdown_connections 2023-10-30T20:24:09.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.902+0000 7f4a49c83640 1 --2- 172.21.15.69:0/4124229765 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c003970 0x7f4a2c003d70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:09.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.902+0000 7f4a49c83640 1 -- 172.21.15.69:0/4124229765 >> 172.21.15.69:0/4124229765 conn(0x7f4a440b7180 msgr2=0x7f4a440b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:09.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.902+0000 7f4a49c83640 1 -- 172.21.15.69:0/4124229765 shutdown_connections 2023-10-30T20:24:09.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.902+0000 7f4a49c83640 1 -- 172.21.15.69:0/4124229765 wait complete. 2023-10-30T20:24:09.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.903+0000 7f4a49c83640 10 monclient: build_initial_monmap 2023-10-30T20:24:09.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.903+0000 7f4a49c83640 10 monclient: monmap: 2023-10-30T20:24:09.935 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:09.935 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:09.935 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:09.936 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:09.936 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:09.936 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:09.936 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:09.936 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:09.936 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:09.937 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f4a49c83640 1 librados: starting msgr at 2023-10-30T20:24:09.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f4a49c83640 1 librados: starting objecter 2023-10-30T20:24:09.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f4a49c83640 1 Processor -- start 2023-10-30T20:24:09.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f4a49c83640 1 -- start start 2023-10-30T20:24:09.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f4a49c83640 1 librados: setting wanted keys 2023-10-30T20:24:09.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f4a49c83640 1 librados: calling monclient init 2023-10-30T20:24:09.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f4a49c83640 10 monclient: init 2023-10-30T20:24:09.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f47997fa640 1 -- 172.21.15.69:0/4109089910 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790033c80 con 0x7f479c0a8e40 2023-10-30T20:24:09.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f47997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f47997fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f47997fa640 10 monclient: dump: 2023-10-30T20:24:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.941 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.941 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.941 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.941 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.941 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.942 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.942 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f47997fa640 10 monclient: _finish_auth 0 2023-10-30T20:24:09.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f47997fa640 10 monclient: _check_auth_tickets 2023-10-30T20:24:09.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f47997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:09.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f47997fa640 1 -- 172.21.15.69:0/4109089910 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4790033df0 con 0x7f479c0a8e40 2023-10-30T20:24:09.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f47a2bbb640 5 monclient: authenticate success, global_id 4357 2023-10-30T20:24:09.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f47a2bbb640 10 monclient: _renew_subs 2023-10-30T20:24:09.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.904+0000 7f47a2bbb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:09.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.905+0000 7f47a2bbb640 1 -- 172.21.15.69:0/4109089910 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f479c137400 con 0x7f479c0a8e40 2023-10-30T20:24:09.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.905+0000 7f47997fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:09.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.905+0000 7f47997fa640 1 -- 172.21.15.69:0/4109089910 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479003d6b0 con 0x7f479c0a8e40 2023-10-30T20:24:09.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.905+0000 7f47997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:09.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.905+0000 7f47997fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:09.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.905+0000 7f47997fa640 10 monclient: dump: 2023-10-30T20:24:09.944 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:09.945 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.945 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.945 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:09.945 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:09.945 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.945 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:09.946 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.905+0000 7f47a2bbb640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.905+0000 7f479affd640 4 set_mon_vals no callback set 2023-10-30T20:24:09.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.906+0000 7f47a2bbb640 10 monclient: _renew_subs 2023-10-30T20:24:09.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.906+0000 7f47a2bbb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:09.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.906+0000 7f47a2bbb640 1 -- 172.21.15.69:0/4109089910 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479c137910 con 0x7f479c0a8e40 2023-10-30T20:24:09.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.906+0000 7f47a2bbb640 1 librados: init done 2023-10-30T20:24:09.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.906+0000 7f4a49c83640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:09.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.906+0000 7f4a49c83640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T20:24:09.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.906+0000 7f4a49c83640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c003970 0x7f4a44144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.906+0000 7f4a49c83640 10 monclient(hunting): picked mon.noname-b con 0x7f4a2c003970 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:09.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.906+0000 7f4a49c83640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a44143930 0x7f4a441445c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.906+0000 7f4a49c83640 10 monclient(hunting): picked mon.noname-a con 0x7f4a44143930 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:09.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.906+0000 7f4a49c83640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a441d6090 0x7f4a441d6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.907+0000 7f4a49c83640 10 monclient(hunting): picked mon.noname-c con 0x7f4a441d6090 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:09.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.907+0000 7f4a49c83640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.907+0000 7f4a49c83640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440b2b10 con 0x7f4a44143930 2023-10-30T20:24:09.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.907+0000 7f4a49c83640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.907+0000 7f4a49c83640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a441d6a50 con 0x7f4a2c003970 2023-10-30T20:24:09.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.907+0000 7f4a49c83640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.907+0000 7f4a49c83640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440ad940 con 0x7f4a441d6090 2023-10-30T20:24:09.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.907+0000 7f4a49c83640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:09.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.907+0000 7f4a49c83640 10 monclient(hunting): authenticate will time out at 1573.196892s 2023-10-30T20:24:09.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.907+0000 7f47a2bbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:09.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.907+0000 7f47a2bbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:09.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.907+0000 7f47a2bbb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:09.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.907+0000 7f47a2bbb640 1 -- 172.21.15.69:0/4109089910 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f479c13c600 con 0x7f479c0a8e40 2023-10-30T20:24:09.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.912+0000 7f4a437fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c003970 0x7f4a44144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.924+0000 7f47997fa640 1 -- 172.21.15.69:0/4109089910 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f479003a040 con 0x7f479c0a8e40 2023-10-30T20:24:09.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.925+0000 7f47997fa640 1 --2- 172.21.15.69:0/4109089910 >> v2:172.21.15.69:6800/1876307353 conn(0x7f478006c7f0 0x7f478006ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.925+0000 7f479bfff640 1 --2- 172.21.15.69:0/4109089910 >> v2:172.21.15.69:6800/1876307353 conn(0x7f478006c7f0 0x7f478006ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.925+0000 7f479bfff640 10 monclient: get_auth_request con 0x7f478006c7f0 auth_method 0 2023-10-30T20:24:09.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.926+0000 7f479bfff640 1 --2- 172.21.15.69:0/4109089910 >> v2:172.21.15.69:6800/1876307353 conn(0x7f478006c7f0 0x7f478006ecb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f479c13dbb0 tx=0x7f4788006c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:09.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.926+0000 7f47997fa640 10 client.4357.objecter ms_handle_connect 0x7f478006c7f0 2023-10-30T20:24:09.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.935+0000 7f530494e640 10 monclient: get_monmap_and_config 2023-10-30T20:24:09.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.935+0000 7f530494e640 10 monclient: build_initial_monmap 2023-10-30T20:24:09.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.935+0000 7f530494e640 10 monclient: monmap: 2023-10-30T20:24:09.955 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:09.956 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.956 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:09.937878+0000 2023-10-30T20:24:09.956 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:09.937878+0000 2023-10-30T20:24:09.956 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:09.956 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.956 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:09.957 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:09.957 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:09.957 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:09.957 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:09.957 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.958 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.936+0000 7f530494e640 1 Processor -- start 2023-10-30T20:24:09.958 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.936+0000 7f530494e640 1 -- start start 2023-10-30T20:24:09.958 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.936+0000 7f530494e640 10 monclient: init 2023-10-30T20:24:09.958 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.937+0000 7f530494e640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:09.958 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.937+0000 7f530494e640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-10-30T20:24:09.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.937+0000 7f530494e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53000a8e40 addr v1:172.21.15.69:6789/0 2023-10-30T20:24:09.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.937+0000 7f530494e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53000b13c0 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:09.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.937+0000 7f530494e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f53001425c0 0x7f530014d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.937+0000 7f530494e640 10 monclient(hunting): picked mon.noname-c con 0x7f53001425c0 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:09.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.937+0000 7f530494e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53000b1b00 con 0x7f53000a8e40 2023-10-30T20:24:09.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.937+0000 7f530494e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53001429d0 con 0x7f53000b13c0 2023-10-30T20:24:09.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.937+0000 7f530494e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.937+0000 7f530494e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53000679a0 con 0x7f53001425c0 2023-10-30T20:24:09.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.937+0000 7f530494e640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:09.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.937+0000 7f530494e640 10 monclient(hunting): authenticate will time out at 1573.227254s 2023-10-30T20:24:09.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.955+0000 7f0b7ccb4640 10 monclient: get_monmap_and_config 2023-10-30T20:24:09.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.955+0000 7f0b7ccb4640 10 monclient: build_initial_monmap 2023-10-30T20:24:09.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.955+0000 7f0b7ccb4640 10 monclient: monmap: 2023-10-30T20:24:09.963 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:09.963 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:09.963 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:09.957459+0000 2023-10-30T20:24:09.963 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:09.957459+0000 2023-10-30T20:24:09.963 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:09.964 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:09.964 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:09.964 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:09.964 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:09.964 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:09.964 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:09.965 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:09.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.956+0000 7f0b7ccb4640 1 Processor -- start 2023-10-30T20:24:09.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.956+0000 7f0b7ccb4640 1 -- start start 2023-10-30T20:24:09.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.956+0000 7f0b7ccb4640 10 monclient: init 2023-10-30T20:24:09.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.956+0000 7f0b7ccb4640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:09.966 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.956+0000 7f0b7ccb4640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-10-30T20:24:09.966 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.956+0000 7f0b7ccb4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b780b4350 0x7f0b780a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.966 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.956+0000 7f0b7ccb4640 10 monclient(hunting): picked mon.noname-b con 0x7f0b780b4350 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:09.966 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.956+0000 7f0b7ccb4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b7813eb50 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:09.966 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.956+0000 7f0b7ccb4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0b78144f50 0x7f0b781453e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.957+0000 7f0b7ccb4640 10 monclient(hunting): picked mon.noname-c con 0x7f0b78144f50 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:09.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.957+0000 7f0b7ccb4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b780aa6e0 con 0x7f0b7813eb50 2023-10-30T20:24:09.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.957+0000 7f0b7ccb4640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.957+0000 7f0b7ccb4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b780679f0 con 0x7f0b78144f50 2023-10-30T20:24:09.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.957+0000 7f0b7ccb4640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:09.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.957+0000 7f0b7ccb4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780b4730 con 0x7f0b780b4350 2023-10-30T20:24:09.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.957+0000 7f0b7ccb4640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:09.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.957+0000 7f0b7ccb4640 10 monclient(hunting): authenticate will time out at 1573.246944s 2023-10-30T20:24:09.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.993+0000 7ff749ffb640 1 -- 172.21.15.69:0/4234578419 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7ff744019450 con 0x7ff7541423f0 2023-10-30T20:24:09.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.993+0000 7ff749ffb640 1 --2- 172.21.15.69:0/4234578419 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff73806c7c0 0x7ff73806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:09.996 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.993+0000 7ff75954c640 1 --2- 172.21.15.69:0/4234578419 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff73806c7c0 0x7ff73806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:09.996 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.993+0000 7ff75954c640 10 monclient: get_auth_request con 0x7ff73806c7c0 auth_method 0 2023-10-30T20:24:09.997 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.994+0000 7ff75954c640 1 --2- 172.21.15.69:0/4234578419 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff73806c7c0 0x7ff73806ec80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff7500015f0 tx=0x7ff750007a40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:09.997 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:09.994+0000 7ff749ffb640 10 client.4338.objecter ms_handle_connect 0x7ff73806c7c0 2023-10-30T20:24:10.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.008+0000 7ff749ffb640 1 -- 172.21.15.69:0/4234578419 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7ff7440584e0 con 0x7ff7541423f0 2023-10-30T20:24:10.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.008+0000 7ff749ffb640 10 client.4338.objecter ms_dispatch 0x7ff754000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:10.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.008+0000 7ff749ffb640 3 client.4338.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:10.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.008+0000 7ff749ffb640 3 client.4338.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:10.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.008+0000 7ff749ffb640 20 client.4338.objecter dump_active .. 0 homeless 2023-10-30T20:24:10.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.008+0000 7ff749ffb640 1 -- 172.21.15.69:0/4234578419 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff74405c580 con 0x7ff7541423f0 2023-10-30T20:24:10.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.008+0000 7ff749ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:10.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.008+0000 7ff749ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:10.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.016+0000 7f4a42ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a44143930 0x7f4a441445c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.017+0000 7f4a42ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a44143930 0x7f4a441445c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56722/0 (socket says 172.21.15.69:56722) 2023-10-30T20:24:10.020 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.017+0000 7f4a42ffd640 1 -- 172.21.15.69:0/1337279337 learned_addr learned my addr 172.21.15.69:0/1337279337 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:10.020 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.017+0000 7f4a42ffd640 10 monclient(hunting): get_auth_request con 0x7f4a44143930 auth_method 0 2023-10-30T20:24:10.020 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.017+0000 7f4a42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:10.020 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.017+0000 7f4a42ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.017+0000 7f4a42ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.018+0000 7f4a42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.019+0000 7f4a42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:10.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.019+0000 7f4a42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:10.026 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.024+0000 7f4a42ffd640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2023-10-30T20:24:10.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.024+0000 7f4a42ffd640 1 -- 172.21.15.69:0/1337279337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a441d6090 msgr2=0x7f4a441d6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.024+0000 7f4a42ffd640 1 --2- 172.21.15.69:0/1337279337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a441d6090 0x7f4a441d6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.024+0000 7f4a42ffd640 1 -- 172.21.15.69:0/1337279337 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c003970 msgr2=0x7f4a44144080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.024+0000 7f4a42ffd640 1 --2- 172.21.15.69:0/1337279337 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c003970 0x7f4a44144080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.024+0000 7f4a42ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:24:10.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.024+0000 7f4a42ffd640 1 monclient: found mon.noname-a 2023-10-30T20:24:10.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.024+0000 7f4a42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:10.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.024+0000 7f4a42ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:10.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.024+0000 7f4a42ffd640 1 -- 172.21.15.69:0/1337279337 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a441d5bb0 con 0x7f4a44143930 2023-10-30T20:24:10.031 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.028+0000 7f4a42ffd640 1 --2- 172.21.15.69:0/1337279337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a44143930 0x7f4a441445c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800c7c0 tx=0x7f4a3800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:10.031 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.028+0000 7f4a2bfff640 10 client.?.objecter ms_handle_connect 0x7f4a44143930 2023-10-30T20:24:10.031 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.028+0000 7f4a2bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:10.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.034+0000 7f52fe59c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f53000a8e40 0x7f530013efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36884/0 (socket says 172.21.15.69:36884) 2023-10-30T20:24:10.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.034+0000 7f52fe59c640 1 -- 172.21.15.69:0/357992496 learned_addr learned my addr 172.21.15.69:0/357992496 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:24:10.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.037+0000 7f4a2bfff640 1 -- 172.21.15.69:0/1337279337 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a380074e0 con 0x7f4a44143930 2023-10-30T20:24:10.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.037+0000 7f4a2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:10.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.037+0000 7f4a2bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:10.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.037+0000 7f4a2bfff640 10 monclient: dump: 2023-10-30T20:24:10.041 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:10.042 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.042 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.042 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.042 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:10.042 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.043 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:10.043 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:10.043 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:10.043 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.037+0000 7f4a2bfff640 10 monclient: _finish_auth 0 2023-10-30T20:24:10.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.037+0000 7f4a2bfff640 10 monclient: _check_auth_tickets 2023-10-30T20:24:10.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.037+0000 7f4a2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:10.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.037+0000 7f4a2bfff640 1 -- 172.21.15.69:0/1337279337 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3800e040 con 0x7f4a44143930 2023-10-30T20:24:10.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.037+0000 7f4a49c83640 5 monclient: authenticate success, global_id 4341 2023-10-30T20:24:10.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.037+0000 7f4a49c83640 10 monclient: _renew_subs 2023-10-30T20:24:10.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.037+0000 7f4a49c83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:10.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.037+0000 7f4a49c83640 1 -- 172.21.15.69:0/1337279337 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a440abd00 con 0x7f4a44143930 2023-10-30T20:24:10.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.038+0000 7f4a2bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:10.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.038+0000 7f4a2bfff640 1 -- 172.21.15.69:0/1337279337 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38004a20 con 0x7f4a44143930 2023-10-30T20:24:10.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.038+0000 7f4a2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:10.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.038+0000 7f4a2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:10.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.038+0000 7f4a2bfff640 10 monclient: dump: 2023-10-30T20:24:10.046 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:10.046 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.046 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.046 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.047 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:10.047 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.047 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:10.047 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:10.047 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:10.048 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.038+0000 7f4a41ffb640 4 set_mon_vals no callback set 2023-10-30T20:24:10.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.038+0000 7f4a49c83640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:10.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.038+0000 7f4a49c83640 10 monclient: _renew_subs 2023-10-30T20:24:10.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.038+0000 7f4a49c83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:10.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.038+0000 7f4a49c83640 1 -- 172.21.15.69:0/1337279337 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a4413b5b0 con 0x7f4a44143930 2023-10-30T20:24:10.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.038+0000 7f4a49c83640 1 librados: init done 2023-10-30T20:24:10.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.039+0000 7f4a49c83640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:10.050 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.039+0000 7f4a49c83640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:10.050 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.039+0000 7f4a49c83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:10.050 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.039+0000 7f4a49c83640 1 -- 172.21.15.69:0/1337279337 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a440a9220 con 0x7f4a44143930 2023-10-30T20:24:10.050 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.045+0000 7f52fd59a640 1 -- 172.21.15.69:0/357992496 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281841324 0 0) 0x7f52f0001110 con 0x7f53000a8e40 2023-10-30T20:24:10.051 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.045+0000 7f52fd59a640 10 monclient(hunting): _negotiate 2023-10-30T20:24:10.051 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.045+0000 7f52fd59a640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:10.051 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.045+0000 7f52fd59a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:10.051 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.045+0000 7f52fd59a640 10 monclient(hunting): my global_id is 4344 2023-10-30T20:24:10.051 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.045+0000 7f52fd59a640 1 -- 172.21.15.69:0/357992496 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e8003520 con 0x7f53000a8e40 2023-10-30T20:24:10.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.069+0000 7f4a2bfff640 1 -- 172.21.15.69:0/1337279337 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f4a38004b90 con 0x7f4a44143930 2023-10-30T20:24:10.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.070+0000 7f4a2bfff640 1 --2- 172.21.15.69:0/1337279337 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4a3006c7c0 0x7f4a3006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.070+0000 7f4a437fe640 1 --2- 172.21.15.69:0/1337279337 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4a3006c7c0 0x7f4a3006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.070+0000 7f4a437fe640 10 monclient: get_auth_request con 0x7f4a3006c7c0 auth_method 0 2023-10-30T20:24:10.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.070+0000 7f4a2bfff640 10 client.4341.objecter ms_handle_connect 0x7f4a3006c7c0 2023-10-30T20:24:10.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.070+0000 7f4a437fe640 1 --2- 172.21.15.69:0/1337279337 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4a3006c7c0 0x7f4a3006ec80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4a34034000 tx=0x7f4a34008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:10.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.074+0000 7f4a2bfff640 1 -- 172.21.15.69:0/1337279337 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f4a38022080 con 0x7f4a44143930 2023-10-30T20:24:10.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.074+0000 7f4a2bfff640 10 client.4341.objecter ms_dispatch 0x7f4a44000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:10.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.074+0000 7f4a2bfff640 3 client.4341.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:10.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.074+0000 7f4a2bfff640 3 client.4341.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:10.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.074+0000 7f4a2bfff640 20 client.4341.objecter dump_active .. 0 homeless 2023-10-30T20:24:10.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.122+0000 7fe452ffd640 1 -- 172.21.15.69:0/590401767 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fe4540044e0 con 0x7fe45c0ab5c0 2023-10-30T20:24:10.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.122+0000 7fe452ffd640 1 --2- 172.21.15.69:0/590401767 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe44006c810 0x7fe44006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.122+0000 7fe462211640 1 --2- 172.21.15.69:0/590401767 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe44006c810 0x7fe44006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.122+0000 7fe462211640 10 monclient: get_auth_request con 0x7fe44006c810 auth_method 0 2023-10-30T20:24:10.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.123+0000 7fe452ffd640 10 client.4316.objecter ms_handle_connect 0x7fe44006c810 2023-10-30T20:24:10.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.123+0000 7fe462211640 1 --2- 172.21.15.69:0/590401767 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe44006c810 0x7fe44006ecd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe458001560 tx=0x7fe458007990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:10.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.128+0000 7f8c108af640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T20:24:10.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.128+0000 7f8c108af640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T20:24:10.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.128+0000 7f8c108af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:10.131 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.128+0000 7f8c108af640 1 -- 172.21.15.69:0/3851739655 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8c0c0aca80 con 0x7f8c0c0a8e40 2023-10-30T20:24:10.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.144+0000 7fe452ffd640 1 -- 172.21.15.69:0/590401767 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fe45407b0f0 con 0x7fe45c0ab5c0 2023-10-30T20:24:10.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.144+0000 7fe452ffd640 10 client.4316.objecter ms_dispatch 0x7fe45c000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:10.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.144+0000 7fe452ffd640 3 client.4316.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:10.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.144+0000 7fe452ffd640 3 client.4316.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:10.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.144+0000 7fe452ffd640 20 client.4316.objecter dump_active .. 0 homeless 2023-10-30T20:24:10.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.144+0000 7fe452ffd640 1 -- 172.21.15.69:0/590401767 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe45407a9f0 con 0x7fe45c0ab5c0 2023-10-30T20:24:10.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.144+0000 7fe452ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:10.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.144+0000 7fe452ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:10.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.147+0000 7f47997fa640 1 -- 172.21.15.69:0/4109089910 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f479007a680 con 0x7f479c0a8e40 2023-10-30T20:24:10.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.147+0000 7f47997fa640 10 client.4357.objecter ms_dispatch 0x7f479c000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:10.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.147+0000 7f47997fa640 3 client.4357.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:10.150 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.147+0000 7f47997fa640 3 client.4357.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:10.150 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.147+0000 7f47997fa640 20 client.4357.objecter dump_active .. 0 homeless 2023-10-30T20:24:10.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.149+0000 7f0b7659c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b780b4350 0x7f0b780a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.149+0000 7f0b7659c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b780b4350 0x7f0b780a83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:40676/0 (socket says 172.21.15.69:40676) 2023-10-30T20:24:10.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.149+0000 7f0b7659c640 1 -- 172.21.15.69:0/1818132008 learned_addr learned my addr 172.21.15.69:0/1818132008 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:10.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.149+0000 7f0b7659c640 10 monclient(hunting): get_auth_request con 0x7f0b780b4350 auth_method 0 2023-10-30T20:24:10.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.149+0000 7f0b7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:10.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.149+0000 7f0b7659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:10.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.149+0000 7f0b7659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:10.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.152+0000 7f0b7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:10.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.152+0000 7f0b7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:10.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.152+0000 7f0b7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:10.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.155+0000 7f0b7659c640 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2023-10-30T20:24:10.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.155+0000 7f0b7659c640 1 -- 172.21.15.69:0/1818132008 >> v2:172.21.15.183:3300/0 conn(0x7f0b78144f50 msgr2=0x7f0b781453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:10.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.155+0000 7f0b7659c640 1 --2- 172.21.15.69:0/1818132008 >> v2:172.21.15.183:3300/0 conn(0x7f0b78144f50 0x7f0b781453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.155+0000 7f0b7659c640 1 -- 172.21.15.69:0/1818132008 >> v1:172.21.15.183:6789/0 conn(0x7f0b7813eb50 legacy=0x7f0b7813ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:10.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.156+0000 7f0b7659c640 10 monclient: _finish_hunting 0 2023-10-30T20:24:10.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.156+0000 7f0b7659c640 1 monclient: found mon.noname-b 2023-10-30T20:24:10.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.156+0000 7f0b7659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.156+0000 7f0b7659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:24:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.156+0000 7f0b7659c640 1 -- 172.21.15.69:0/1818132008 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7813f680 con 0x7f0b780b4350 2023-10-30T20:24:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.156+0000 7f47997fa640 1 -- 172.21.15.69:0/4109089910 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f479002f050 con 0x7f479c0a8e40 2023-10-30T20:24:10.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.156+0000 7f47997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:10.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.156+0000 7f47997fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.163+0000 7f0b7659c640 1 --2- 172.21.15.69:0/1818132008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b780b4350 0x7f0b780a83d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009870 tx=0x7f0b6802b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=451772df7890e093 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:10.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.169+0000 7f0b7559a640 1 -- 172.21.15.69:0/1818132008 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68031740 con 0x7f0b780b4350 2023-10-30T20:24:10.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.169+0000 7f0b7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:10.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.169+0000 7f0b7559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:10.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.169+0000 7f0b7559a640 10 monclient: dump: 2023-10-30T20:24:10.173 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:10.173 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.174 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.174 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.174 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:10.174 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.175 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:10.175 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:10.175 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:10.175 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.169+0000 7f0b7559a640 10 monclient: _finish_auth 0 2023-10-30T20:24:10.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.169+0000 7f0b7559a640 10 monclient: _check_auth_tickets 2023-10-30T20:24:10.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.169+0000 7f0b7559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:10.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.169+0000 7f0b7ccb4640 5 monclient: authenticate success, global_id 4319 2023-10-30T20:24:10.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.169+0000 7f0b7ccb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:10.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.169+0000 7f0b7559a640 1 -- 172.21.15.69:0/1818132008 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b680318b0 con 0x7f0b780b4350 2023-10-30T20:24:10.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.169+0000 7f0b7559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:10.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.169+0000 7f0b7ccb4640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:10.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.169+0000 7f0b7ccb4640 4 set_mon_vals no callback set 2023-10-30T20:24:10.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.170+0000 7f0b7559a640 1 -- 172.21.15.69:0/1818132008 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68031740 con 0x7f0b780b4350 2023-10-30T20:24:10.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.170+0000 7f0b7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:10.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.170+0000 7f0b7559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:10.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.170+0000 7f0b7559a640 10 monclient: dump: 2023-10-30T20:24:10.178 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:10.178 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.179 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.179 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.179 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:10.179 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.179 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:10.180 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:10.180 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:10.180 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.170+0000 7f0b7ccb4640 10 monclient: shutdown 2023-10-30T20:24:10.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.170+0000 7f0b7ccb4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:10.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.170+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/1818132008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b780b4350 msgr2=0x7f0b780a83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.170+0000 7f0b7ccb4640 1 --2- 172.21.15.69:0/1818132008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b780b4350 0x7f0b780a83d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009870 tx=0x7f0b6802b7e0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.171+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/1818132008 shutdown_connections 2023-10-30T20:24:10.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.171+0000 7f0b7ccb4640 1 --2- 172.21.15.69:0/1818132008 >> v2:172.21.15.183:3300/0 conn(0x7f0b78144f50 0x7f0b781453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.171+0000 7f0b7ccb4640 1 --2- 172.21.15.69:0/1818132008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b780b4350 0x7f0b780a83d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.171+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/1818132008 >> 172.21.15.69:0/1818132008 conn(0x7f0b78065ea0 msgr2=0x7f0b780662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:10.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.171+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/1818132008 shutdown_connections 2023-10-30T20:24:10.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.171+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/1818132008 wait complete. 2023-10-30T20:24:10.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.172+0000 7f0b7ccb4640 10 monclient: build_initial_monmap 2023-10-30T20:24:10.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.172+0000 7f0b7ccb4640 10 monclient: monmap: 2023-10-30T20:24:10.187 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:10.187 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:10.188 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:10.188 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:10.188 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:10.188 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:10.188 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:10.188 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:10.189 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:10.189 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.173+0000 7f0b7ccb4640 1 librados: starting msgr at 2023-10-30T20:24:10.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.173+0000 7f0b7ccb4640 1 librados: starting objecter 2023-10-30T20:24:10.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.173+0000 7f0b7ccb4640 1 Processor -- start 2023-10-30T20:24:10.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.173+0000 7f0b7ccb4640 1 -- start start 2023-10-30T20:24:10.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.173+0000 7f0b7ccb4640 1 librados: setting wanted keys 2023-10-30T20:24:10.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.173+0000 7f0b7ccb4640 1 librados: calling monclient init 2023-10-30T20:24:10.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.173+0000 7f0b7ccb4640 10 monclient: init 2023-10-30T20:24:10.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:10.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:24:10.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b780b4350 0x7f0b780b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 10 monclient(hunting): picked mon.noname-a con 0x7f0b780b4350 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:10.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7813eb50 0x7f0b7813fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 10 monclient(hunting): picked mon.noname-b con 0x7f0b7813eb50 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:10.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b78144f50 0x7f0b781da310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 10 monclient(hunting): picked mon.noname-c con 0x7f0b78144f50 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:10.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:10.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780a90e0 con 0x7f0b780b4350 2023-10-30T20:24:10.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:10.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b781415e0 con 0x7f0b7813eb50 2023-10-30T20:24:10.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:10.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b781d43e0 con 0x7f0b78144f50 2023-10-30T20:24:10.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:10.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.174+0000 7f0b7ccb4640 10 monclient(hunting): authenticate will time out at 1573.464664s 2023-10-30T20:24:10.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.175+0000 7f0b76d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b78144f50 0x7f0b781da310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.176+0000 7f0b76d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b78144f50 0x7f0b781da310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:40686/0 (socket says 172.21.15.69:40686) 2023-10-30T20:24:10.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.176+0000 7f0b76d9d640 1 -- 172.21.15.69:0/2023654174 learned_addr learned my addr 172.21.15.69:0/2023654174 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:10.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.176+0000 7f0b76d9d640 10 monclient(hunting): get_auth_request con 0x7f0b78144f50 auth_method 0 2023-10-30T20:24:10.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.176+0000 7f0b76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:10.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.176+0000 7f0b76d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:10.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.176+0000 7f0b76d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:10.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.176+0000 7f0b76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:10.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.176+0000 7f0b76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.176+0000 7f0b76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.181+0000 7f0b76d9d640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2023-10-30T20:24:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.181+0000 7f0b76d9d640 1 -- 172.21.15.69:0/2023654174 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7813eb50 msgr2=0x7f0b7813fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.181+0000 7f0b76d9d640 1 --2- 172.21.15.69:0/2023654174 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7813eb50 0x7f0b7813fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.181+0000 7f0b76d9d640 1 -- 172.21.15.69:0/2023654174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b780b4350 msgr2=0x7f0b780b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.181+0000 7f0b76d9d640 1 --2- 172.21.15.69:0/2023654174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b780b4350 0x7f0b780b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.181+0000 7f0b76d9d640 10 monclient: _finish_hunting 0 2023-10-30T20:24:10.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.181+0000 7f0b76d9d640 1 monclient: found mon.noname-c 2023-10-30T20:24:10.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.181+0000 7f0b76d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:10.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.181+0000 7f0b76d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:24:10.198 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.181+0000 7f0b76d9d640 1 -- 172.21.15.69:0/2023654174 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b78140250 con 0x7f0b78144f50 2023-10-30T20:24:10.198 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.182+0000 7f52fed9d640 1 --2- 172.21.15.69:0/357992496 >> v2:172.21.15.183:3300/0 conn(0x7f53001425c0 0x7f530014d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.184+0000 7f0b76d9d640 1 --2- 172.21.15.69:0/2023654174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b78144f50 0x7f0b781da310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0b7000b970 tx=0x7f0b7000be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:10.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.184+0000 7f0b66ffd640 10 client.?.objecter ms_handle_connect 0x7f0b78144f50 2023-10-30T20:24:10.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.184+0000 7f0b66ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:10.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.184+0000 7f8c02ffd640 1 -- 172.21.15.69:0/3851739655 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8bfc031890 con 0x7f8c0c0a8e40 2023-10-30T20:24:10.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.184+0000 7f8c02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T20:24:10.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.184+0000 7f8c02ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:10.200 INFO:teuthology.orchestra.run.smithi069.stdout:68719476741 2023-10-30T20:24:10.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.185+0000 7f52fed9d640 10 monclient(hunting): get_auth_request con 0x7f53001425c0 auth_method 0 2023-10-30T20:24:10.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.185+0000 7f52fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:10.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.185+0000 7f52fed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:10.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.185+0000 7f52fed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:10.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.185+0000 7f52fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:10.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.185+0000 7f52fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:10.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.185+0000 7f52fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:10.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 10 librados: watch_flush enter 2023-10-30T20:24:10.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 10 librados: watch_flush exit 2023-10-30T20:24:10.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 20 client.4348.objecter shutdown clearing up homeless session... 2023-10-30T20:24:10.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 10 client.4348.objecter successfully canceled tick 2023-10-30T20:24:10.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 1 -- 172.21.15.69:0/3851739655 >> v2:172.21.15.69:6800/1876307353 conn(0x7f8bec06c7c0 msgr2=0x7f8bec06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 1 --2- 172.21.15.69:0/3851739655 >> v2:172.21.15.69:6800/1876307353 conn(0x7f8bec06c7c0 0x7f8bec06ec80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4001380 tx=0x7f8bf4007b90 comp rx=0 tx=0).stop 2023-10-30T20:24:10.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 10 monclient: shutdown 2023-10-30T20:24:10.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:10.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 1 -- 172.21.15.69:0/3851739655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c0c0a8e40 msgr2=0x7f8c0c0ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 1 --2- 172.21.15.69:0/3851739655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c0c0a8e40 0x7f8c0c0ad930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc034040 tx=0x7f8bfc0046c0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 1 -- 172.21.15.69:0/3851739655 shutdown_connections 2023-10-30T20:24:10.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 1 --2- 172.21.15.69:0/3851739655 >> v2:172.21.15.69:6800/1876307353 conn(0x7f8bec06c7c0 0x7f8bec06ec80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 1 --2- 172.21.15.69:0/3851739655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0c145770 0x7f8c0c1d3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b66ffd640 1 -- 172.21.15.69:0/2023654174 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7000cb40 con 0x7f0b78144f50 2023-10-30T20:24:10.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:10.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b66ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:10.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b66ffd640 10 monclient: dump: 2023-10-30T20:24:10.207 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:10.208 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.208 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.208 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.208 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:10.208 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.209 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:10.209 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:10.209 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:10.209 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.209 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b66ffd640 10 monclient: _finish_auth 0 2023-10-30T20:24:10.210 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b66ffd640 10 monclient: _check_auth_tickets 2023-10-30T20:24:10.210 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:10.211 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 1 --2- 172.21.15.69:0/3851739655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0c1da150 0x7f8c0c1432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.211 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 1 --2- 172.21.15.69:0/3851739655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c0c0a8e40 0x7f8c0c0ad930 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.211 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 1 -- 172.21.15.69:0/3851739655 >> 172.21.15.69:0/3851739655 conn(0x7f8c0c0b7180 msgr2=0x7f8c0c0b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:10.211 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 1 -- 172.21.15.69:0/3851739655 shutdown_connections 2023-10-30T20:24:10.212 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.187+0000 7f8c108af640 1 -- 172.21.15.69:0/3851739655 wait complete. 2023-10-30T20:24:10.212 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f8c108af640 1 librados: shutdown 2023-10-30T20:24:10.213 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b7ccb4640 5 monclient: authenticate success, global_id 4322 2023-10-30T20:24:10.213 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b7ccb4640 10 monclient: _renew_subs 2023-10-30T20:24:10.213 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b7ccb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:10.213 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/2023654174 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b780afd50 con 0x7f0b78144f50 2023-10-30T20:24:10.213 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b7ccb4640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:10.214 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b7ccb4640 10 monclient: _renew_subs 2023-10-30T20:24:10.214 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b7ccb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:10.214 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/2023654174 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b78141b00 con 0x7f0b78144f50 2023-10-30T20:24:10.214 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.188+0000 7f0b7ccb4640 1 librados: init done 2023-10-30T20:24:10.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.189+0000 7f52fed9d640 10 monclient(hunting): handle_auth_done global_id 4360 payload 274 2023-10-30T20:24:10.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.189+0000 7f52fed9d640 1 -- 172.21.15.69:0/357992496 >> v1:172.21.15.183:6789/0 conn(0x7f53000b13c0 legacy=0x7f5300141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.189+0000 7f52fed9d640 1 -- 172.21.15.69:0/357992496 >> v1:172.21.15.69:6789/0 conn(0x7f53000a8e40 legacy=0x7f530013efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.189+0000 7f52fed9d640 10 monclient: _finish_hunting 0 2023-10-30T20:24:10.215 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.189+0000 7f52fed9d640 1 monclient: found mon.noname-c 2023-10-30T20:24:10.216 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.189+0000 7f52fed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:10.216 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.189+0000 7f52fed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:24:10.216 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.189+0000 7f52fed9d640 1 -- 172.21.15.69:0/357992496 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53000a8920 con 0x7f53001425c0 2023-10-30T20:24:10.217 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.190+0000 7f0b7ccb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:10.217 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.190+0000 7f0b7ccb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:10.217 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.190+0000 7f0b7ccb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:10.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.190+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/2023654174 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b781d9f70 con 0x7f0b78144f50 2023-10-30T20:24:10.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.191+0000 7f0b66ffd640 1 -- 172.21.15.69:0/2023654174 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7000ccb0 con 0x7f0b78144f50 2023-10-30T20:24:10.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.191+0000 7f0b66ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:10.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.191+0000 7f0b66ffd640 1 -- 172.21.15.69:0/2023654174 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7001e640 con 0x7f0b78144f50 2023-10-30T20:24:10.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.191+0000 7f0b66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:10.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.191+0000 7f0b66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:10.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.191+0000 7f0b66ffd640 10 monclient: dump: 2023-10-30T20:24:10.219 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:10.219 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.219 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.219 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.220 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:10.220 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.220 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:10.220 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:10.220 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:10.220 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.191+0000 7f0b74d99640 4 set_mon_vals no callback set 2023-10-30T20:24:10.223 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.194+0000 7f52fed9d640 1 --2- 172.21.15.69:0/357992496 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53001425c0 0x7f530014d330 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009d90 tx=0x7f52f802be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7404ebb00e0bc25 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:10.223 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.196+0000 7f52fd59a640 1 -- 172.21.15.69:0/357992496 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f802cbe0 con 0x7f53001425c0 2023-10-30T20:24:10.223 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.196+0000 7f52fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:10.223 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.196+0000 7f52fd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:10.224 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.196+0000 7f52fd59a640 10 monclient: dump: 2023-10-30T20:24:10.224 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:10.224 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.224 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.224 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.225 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:10.225 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.225 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:10.225 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:10.225 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:10.225 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.226 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.196+0000 7f52fd59a640 10 monclient: _finish_auth 0 2023-10-30T20:24:10.226 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.196+0000 7f52fd59a640 10 monclient: _check_auth_tickets 2023-10-30T20:24:10.226 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.196+0000 7f52fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:10.226 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.196+0000 7f530494e640 5 monclient: authenticate success, global_id 4360 2023-10-30T20:24:10.226 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.196+0000 7f530494e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:10.227 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.198+0000 7f52fd59a640 1 -- 172.21.15.69:0/357992496 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f802cd50 con 0x7f53001425c0 2023-10-30T20:24:10.227 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.198+0000 7f52fd59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:10.227 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.198+0000 7f52fd59a640 1 -- 172.21.15.69:0/357992496 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f8034640 con 0x7f53001425c0 2023-10-30T20:24:10.227 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.198+0000 7f52fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:10.227 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.198+0000 7f52fd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:10.228 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.198+0000 7f52fd59a640 10 monclient: dump: 2023-10-30T20:24:10.228 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:10.228 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.228 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.228 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.229 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:10.229 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.229 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:10.229 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:10.229 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:10.229 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.198+0000 7f530494e640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.198+0000 7f530494e640 4 set_mon_vals no callback set 2023-10-30T20:24:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.199+0000 7f530494e640 10 monclient: shutdown 2023-10-30T20:24:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.199+0000 7f530494e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.199+0000 7f530494e640 1 -- 172.21.15.69:0/357992496 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53001425c0 msgr2=0x7f530014d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.199+0000 7f530494e640 1 --2- 172.21.15.69:0/357992496 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53001425c0 0x7f530014d330 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009d90 tx=0x7f52f802be30 comp rx=0 tx=0).stop 2023-10-30T20:24:10.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.199+0000 7f530494e640 1 -- 172.21.15.69:0/357992496 shutdown_connections 2023-10-30T20:24:10.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.199+0000 7f530494e640 1 --2- 172.21.15.69:0/357992496 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53001425c0 0x7f530014d330 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.200+0000 7f530494e640 1 -- 172.21.15.69:0/357992496 >> 172.21.15.69:0/357992496 conn(0x7f53000b7180 msgr2=0x7f53000b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:10.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.200+0000 7f530494e640 1 -- 172.21.15.69:0/357992496 shutdown_connections 2023-10-30T20:24:10.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.200+0000 7f530494e640 1 -- 172.21.15.69:0/357992496 wait complete. 2023-10-30T20:24:10.232 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.201+0000 7f530494e640 10 monclient: build_initial_monmap 2023-10-30T20:24:10.232 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.201+0000 7f530494e640 10 monclient: monmap: 2023-10-30T20:24:10.232 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:10.232 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:10.232 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:10.233 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:10.233 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:10.233 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:10.233 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:10.233 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:10.234 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:10.234 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.234 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.202+0000 7f530494e640 1 librados: starting msgr at 2023-10-30T20:24:10.234 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.202+0000 7f530494e640 1 librados: starting objecter 2023-10-30T20:24:10.234 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.202+0000 7f530494e640 1 Processor -- start 2023-10-30T20:24:10.235 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.202+0000 7f530494e640 1 -- start start 2023-10-30T20:24:10.235 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.202+0000 7f530494e640 1 librados: setting wanted keys 2023-10-30T20:24:10.235 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.202+0000 7f530494e640 1 librados: calling monclient init 2023-10-30T20:24:10.235 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.202+0000 7f530494e640 10 monclient: init 2023-10-30T20:24:10.235 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.204+0000 7f530494e640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:10.235 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.204+0000 7f530494e640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T20:24:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.204+0000 7f530494e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53000a8e40 0x7f53001d6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.204+0000 7f530494e640 10 monclient(hunting): picked mon.noname-c con 0x7f53000a8e40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.204+0000 7f530494e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53001425c0 0x7f530013b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.204+0000 7f530494e640 10 monclient(hunting): picked mon.noname-b con 0x7f53001425c0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.204+0000 7f530494e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53001d7680 0x7f530013b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.237 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.204+0000 7f530494e640 10 monclient(hunting): picked mon.noname-a con 0x7f53001d7680 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:10.237 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.204+0000 7f530494e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:10.237 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.205+0000 7f530494e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53000a8460 con 0x7f53001d7680 2023-10-30T20:24:10.237 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.205+0000 7f530494e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:10.237 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.205+0000 7f530494e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5300143710 con 0x7f53001425c0 2023-10-30T20:24:10.238 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.205+0000 7f530494e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:10.238 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.205+0000 7f530494e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530013bc30 con 0x7f53000a8e40 2023-10-30T20:24:10.238 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.205+0000 7f530494e640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:10.238 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.205+0000 7f530494e640 10 monclient(hunting): authenticate will time out at 1573.494846s 2023-10-30T20:24:10.239 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476741 for osd.1 2023-10-30T20:24:10.240 DEBUG:teuthology.parallel:result is None 2023-10-30T20:24:10.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.205+0000 7f52fdd9b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53001425c0 0x7f530013b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.205+0000 7f52fdd9b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53001425c0 0x7f530013b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:53184/0 (socket says 172.21.15.69:53184) 2023-10-30T20:24:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.205+0000 7f52fdd9b640 1 -- 172.21.15.69:0/1562146012 learned_addr learned my addr 172.21.15.69:0/1562146012 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.205+0000 7f52fdd9b640 10 monclient(hunting): get_auth_request con 0x7f53001425c0 auth_method 0 2023-10-30T20:24:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.205+0000 7f52fdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.205+0000 7f52fdd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.205+0000 7f52fdd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:10.244 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.206+0000 7f52fdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:10.244 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.206+0000 7f52fdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:10.244 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.206+0000 7f52fdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:10.244 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.208+0000 7f52fdd9b640 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2023-10-30T20:24:10.244 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.208+0000 7f52fdd9b640 1 -- 172.21.15.69:0/1562146012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53000a8e40 msgr2=0x7f53001d6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.208+0000 7f52fdd9b640 1 --2- 172.21.15.69:0/1562146012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53000a8e40 0x7f53001d6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.209+0000 7f52fdd9b640 1 -- 172.21.15.69:0/1562146012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53001d7680 msgr2=0x7f530013b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.209+0000 7f52fdd9b640 1 --2- 172.21.15.69:0/1562146012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53001d7680 0x7f530013b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.209+0000 7f52fdd9b640 10 monclient: _finish_hunting 0 2023-10-30T20:24:10.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.209+0000 7f52fdd9b640 1 monclient: found mon.noname-b 2023-10-30T20:24:10.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.209+0000 7f52fdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:10.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.209+0000 7f52fdd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:24:10.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.209+0000 7f52fdd9b640 1 -- 172.21.15.69:0/1562146012 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53000ad790 con 0x7f53001425c0 2023-10-30T20:24:10.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.211+0000 7f52fdd9b640 1 --2- 172.21.15.69:0/1562146012 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53001425c0 0x7f530013b1b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f52f400c440 tx=0x7f52f4001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:10.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.211+0000 7f52eeffd640 10 client.?.objecter ms_handle_connect 0x7f53001425c0 2023-10-30T20:24:10.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.211+0000 7f52eeffd640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:10.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f52eeffd640 1 -- 172.21.15.69:0/1562146012 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f40052d0 con 0x7f53001425c0 2023-10-30T20:24:10.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f52eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:10.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f52eeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:10.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f52eeffd640 10 monclient: dump: 2023-10-30T20:24:10.248 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:10.248 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.248 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.248 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.248 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:10.248 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.249 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:10.249 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:10.249 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:10.249 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f52eeffd640 10 monclient: _finish_auth 0 2023-10-30T20:24:10.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f52eeffd640 10 monclient: _check_auth_tickets 2023-10-30T20:24:10.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f52eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:10.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f52eeffd640 1 -- 172.21.15.69:0/1562146012 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f400e040 con 0x7f53001425c0 2023-10-30T20:24:10.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f52eeffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:10.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f52eeffd640 1 -- 172.21.15.69:0/1562146012 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f4007260 con 0x7f53001425c0 2023-10-30T20:24:10.251 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f52eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:10.251 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f52eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:24:10.251 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f52eeffd640 10 monclient: dump: 2023-10-30T20:24:10.251 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:10.251 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.252 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.252 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.252 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:10.252 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.252 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:10.253 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:10.253 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:10.253 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.253 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f530494e640 5 monclient: authenticate success, global_id 4363 2023-10-30T20:24:10.253 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f52fcd99640 4 set_mon_vals no callback set 2023-10-30T20:24:10.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f530494e640 10 monclient: _renew_subs 2023-10-30T20:24:10.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f530494e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:10.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f530494e640 1 -- 172.21.15.69:0/1562146012 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5300143230 con 0x7f53001425c0 2023-10-30T20:24:10.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f530494e640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:10.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f530494e640 10 monclient: _renew_subs 2023-10-30T20:24:10.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f530494e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f530494e640 1 -- 172.21.15.69:0/1562146012 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5300143d80 con 0x7f53001425c0 2023-10-30T20:24:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.215+0000 7f530494e640 1 librados: init done 2023-10-30T20:24:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.217+0000 7f530494e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.217+0000 7f530494e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.217+0000 7f530494e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:10.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.217+0000 7f530494e640 1 -- 172.21.15.69:0/1562146012 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53000a9220 con 0x7f53001425c0 2023-10-30T20:24:10.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.225+0000 7ff75b7b0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T20:24:10.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.225+0000 7ff75b7b0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T20:24:10.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.225+0000 7ff75b7b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:10.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.225+0000 7ff75b7b0640 1 -- 172.21.15.69:0/4234578419 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff7540ab450 con 0x7ff7541423f0 2023-10-30T20:24:10.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.229+0000 7f0b66ffd640 1 -- 172.21.15.69:0/2023654174 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f0b7001e7b0 con 0x7f0b78144f50 2023-10-30T20:24:10.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.229+0000 7f0b66ffd640 1 --2- 172.21.15.69:0/2023654174 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0b5806c7f0 0x7f0b5806ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.229+0000 7f0b7659c640 1 --2- 172.21.15.69:0/2023654174 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0b5806c7f0 0x7f0b5806ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.229+0000 7f0b7659c640 10 monclient: get_auth_request con 0x7f0b5806c7f0 auth_method 0 2023-10-30T20:24:10.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.230+0000 7f0b7659c640 1 --2- 172.21.15.69:0/2023654174 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0b5806c7f0 0x7f0b5806ecb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0b68035000 tx=0x7f0b68030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:10.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.230+0000 7f0b66ffd640 10 client.4322.objecter ms_handle_connect 0x7f0b5806c7f0 2023-10-30T20:24:10.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.234+0000 7f52eeffd640 1 -- 172.21.15.69:0/1562146012 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f52f40073d0 con 0x7f53001425c0 2023-10-30T20:24:10.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.235+0000 7f52eeffd640 1 --2- 172.21.15.69:0/1562146012 >> v2:172.21.15.69:6800/1876307353 conn(0x7f52e006c7c0 0x7f52e006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.235+0000 7f52fe59c640 1 --2- 172.21.15.69:0/1562146012 >> v2:172.21.15.69:6800/1876307353 conn(0x7f52e006c7c0 0x7f52e006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.235+0000 7f52fe59c640 10 monclient: get_auth_request con 0x7f52e006c7c0 auth_method 0 2023-10-30T20:24:10.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.235+0000 7f52fe59c640 1 --2- 172.21.15.69:0/1562146012 >> v2:172.21.15.69:6800/1876307353 conn(0x7f52e006c7c0 0x7f52e006ec80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f52f0005880 tx=0x7f52f0001150 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:10.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.235+0000 7f52eeffd640 10 client.4363.objecter ms_handle_connect 0x7f52e006c7c0 2023-10-30T20:24:10.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.342+0000 7f4a2bfff640 1 -- 172.21.15.69:0/1337279337 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4a38007650 con 0x7f4a44143930 2023-10-30T20:24:10.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.342+0000 7f4a2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:10.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.342+0000 7f4a2bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:10.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.357+0000 7f47a2bbb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T20:24:10.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.357+0000 7f47a2bbb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T20:24:10.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.357+0000 7f47a2bbb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:10.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.357+0000 7f47a2bbb640 1 -- 172.21.15.69:0/4109089910 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f479c137910 con 0x7f479c0a8e40 2023-10-30T20:24:10.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.367+0000 7fe450ff9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T20:24:10.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.367+0000 7fe450ff9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T20:24:10.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.367+0000 7fe450ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:10.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.367+0000 7fe450ff9640 1 -- 172.21.15.69:0/590401767 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe4380055c0 con 0x7fe45c0ab5c0 2023-10-30T20:24:10.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.445+0000 7ff749ffb640 1 -- 172.21.15.69:0/4234578419 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff744060730 con 0x7ff7541423f0 2023-10-30T20:24:10.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.445+0000 7ff749ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T20:24:10.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.445+0000 7ff749ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:10.448 INFO:teuthology.orchestra.run.smithi069.stdout:55834574855 2023-10-30T20:24:10.450 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 10 librados: watch_flush enter 2023-10-30T20:24:10.450 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 10 librados: watch_flush exit 2023-10-30T20:24:10.450 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 20 client.4338.objecter shutdown clearing up homeless session... 2023-10-30T20:24:10.450 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 10 client.4338.objecter successfully canceled tick 2023-10-30T20:24:10.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 1 -- 172.21.15.69:0/4234578419 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff73806c7c0 msgr2=0x7ff73806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 1 --2- 172.21.15.69:0/4234578419 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff73806c7c0 0x7ff73806ec80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff7500015f0 tx=0x7ff750007a40 comp rx=0 tx=0).stop 2023-10-30T20:24:10.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 10 monclient: shutdown 2023-10-30T20:24:10.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:10.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 1 -- 172.21.15.69:0/4234578419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7541423f0 msgr2=0x7ff7541defd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 1 --2- 172.21.15.69:0/4234578419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7541423f0 0x7ff7541defd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff74400b480 tx=0x7ff74400b940 comp rx=0 tx=0).stop 2023-10-30T20:24:10.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 1 -- 172.21.15.69:0/4234578419 shutdown_connections 2023-10-30T20:24:10.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 1 --2- 172.21.15.69:0/4234578419 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff73806c7c0 0x7ff73806ec80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 1 --2- 172.21.15.69:0/4234578419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7541df510 0x7ff7541db320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 1 --2- 172.21.15.69:0/4234578419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7541423f0 0x7ff7541defd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 1 --2- 172.21.15.69:0/4234578419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7540a8e40 0x7ff7541dea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.453 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 1 -- 172.21.15.69:0/4234578419 >> 172.21.15.69:0/4234578419 conn(0x7ff7540b7180 msgr2=0x7ff7540b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:10.453 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 1 -- 172.21.15.69:0/4234578419 shutdown_connections 2023-10-30T20:24:10.453 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 1 -- 172.21.15.69:0/4234578419 wait complete. 2023-10-30T20:24:10.453 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.447+0000 7ff75b7b0640 1 librados: shutdown 2023-10-30T20:24:10.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.4 2023-10-30T20:24:10.461 DEBUG:teuthology.parallel:result is None 2023-10-30T20:24:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.458+0000 7f52eeffd640 1 -- 172.21.15.69:0/1562146012 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f52f4005480 con 0x7f53001425c0 2023-10-30T20:24:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.458+0000 7f52eeffd640 10 client.4363.objecter ms_dispatch 0x7f5300000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.458+0000 7f52eeffd640 3 client.4363.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.458+0000 7f52eeffd640 3 client.4363.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.458+0000 7f52eeffd640 20 client.4363.objecter dump_active .. 0 homeless 2023-10-30T20:24:10.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.467+0000 7f52eeffd640 1 -- 172.21.15.69:0/1562146012 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f52f4086e50 con 0x7f53001425c0 2023-10-30T20:24:10.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.467+0000 7f52eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:10.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.467+0000 7f52eeffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:10.476 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.474+0000 7f47997fa640 1 -- 172.21.15.69:0/4109089910 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4790031890 con 0x7f479c0a8e40 2023-10-30T20:24:10.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.474+0000 7f47997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T20:24:10.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.474+0000 7f47997fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:10.477 INFO:teuthology.orchestra.run.smithi069.stdout:60129542149 2023-10-30T20:24:10.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 10 librados: watch_flush enter 2023-10-30T20:24:10.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 10 librados: watch_flush exit 2023-10-30T20:24:10.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 20 client.4357.objecter shutdown clearing up homeless session... 2023-10-30T20:24:10.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 10 client.4357.objecter successfully canceled tick 2023-10-30T20:24:10.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 1 -- 172.21.15.69:0/4109089910 >> v2:172.21.15.69:6800/1876307353 conn(0x7f478006c7f0 msgr2=0x7f478006ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 1 --2- 172.21.15.69:0/4109089910 >> v2:172.21.15.69:6800/1876307353 conn(0x7f478006c7f0 0x7f478006ecb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f479c13dbb0 tx=0x7f4788006c90 comp rx=0 tx=0).stop 2023-10-30T20:24:10.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 10 monclient: shutdown 2023-10-30T20:24:10.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:10.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 1 -- 172.21.15.69:0/4109089910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f479c0a8e40 msgr2=0x7f479c0af740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 1 --2- 172.21.15.69:0/4109089910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f479c0a8e40 0x7f479c0af740 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f479002b830 tx=0x7f47900046c0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 1 -- 172.21.15.69:0/4109089910 shutdown_connections 2023-10-30T20:24:10.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 1 --2- 172.21.15.69:0/4109089910 >> v2:172.21.15.69:6800/1876307353 conn(0x7f478006c7f0 0x7f478006ecb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 1 --2- 172.21.15.69:0/4109089910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f479c13c7d0 0x7f479c1de6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 1 --2- 172.21.15.69:0/4109089910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f479c139670 0x7f479c1de180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 1 --2- 172.21.15.69:0/4109089910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f479c0a8e40 0x7f479c0af740 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 1 -- 172.21.15.69:0/4109089910 >> 172.21.15.69:0/4109089910 conn(0x7f479c0b7180 msgr2=0x7f479c0b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:10.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.476+0000 7f47a2bbb640 1 -- 172.21.15.69:0/4109089910 shutdown_connections 2023-10-30T20:24:10.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.477+0000 7f47a2bbb640 1 -- 172.21.15.69:0/4109089910 wait complete. 2023-10-30T20:24:10.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.477+0000 7f47a2bbb640 1 librados: shutdown 2023-10-30T20:24:10.490 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.7 2023-10-30T20:24:10.490 DEBUG:teuthology.parallel:result is None 2023-10-30T20:24:10.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.510+0000 7f0b66ffd640 1 -- 172.21.15.69:0/2023654174 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f0b70058920 con 0x7f0b78144f50 2023-10-30T20:24:10.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.510+0000 7f0b66ffd640 10 client.4322.objecter ms_dispatch 0x7f0b78000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:10.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.510+0000 7f0b66ffd640 3 client.4322.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:10.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.510+0000 7f0b66ffd640 3 client.4322.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:10.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.510+0000 7f0b66ffd640 20 client.4322.objecter dump_active .. 0 homeless 2023-10-30T20:24:10.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.510+0000 7f0b66ffd640 1 -- 172.21.15.69:0/2023654174 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0b7005c9c0 con 0x7f0b78144f50 2023-10-30T20:24:10.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.510+0000 7f0b66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:10.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.510+0000 7f0b66ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:10.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.527+0000 7fe452ffd640 1 -- 172.21.15.69:0/590401767 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe45407eb40 con 0x7fe45c0ab5c0 2023-10-30T20:24:10.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.527+0000 7fe452ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T20:24:10.531 INFO:teuthology.orchestra.run.smithi069.stdout:64424509446 2023-10-30T20:24:10.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.527+0000 7fe452ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:10.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 10 librados: watch_flush enter 2023-10-30T20:24:10.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 10 librados: watch_flush exit 2023-10-30T20:24:10.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 20 client.4316.objecter shutdown clearing up homeless session... 2023-10-30T20:24:10.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 10 client.4316.objecter successfully canceled tick 2023-10-30T20:24:10.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 1 -- 172.21.15.69:0/590401767 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe44006c810 msgr2=0x7fe44006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 1 --2- 172.21.15.69:0/590401767 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe44006c810 0x7fe44006ecd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe458001560 tx=0x7fe458007990 comp rx=0 tx=0).stop 2023-10-30T20:24:10.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 10 monclient: shutdown 2023-10-30T20:24:10.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:10.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 1 -- 172.21.15.69:0/590401767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe45c0ab5c0 msgr2=0x7fe45c0ada50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 1 --2- 172.21.15.69:0/590401767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe45c0ab5c0 0x7fe45c0ada50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe454009f70 tx=0x7fe454004a40 comp rx=0 tx=0).stop 2023-10-30T20:24:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 1 -- 172.21.15.69:0/590401767 shutdown_connections 2023-10-30T20:24:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 1 --2- 172.21.15.69:0/590401767 >> v2:172.21.15.69:6800/1876307353 conn(0x7fe44006c810 0x7fe44006ecd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 1 --2- 172.21.15.69:0/590401767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe45c0ab5c0 0x7fe45c0ada50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 1 --2- 172.21.15.69:0/590401767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe45c1423f0 0x7fe45c0ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 1 --2- 172.21.15.69:0/590401767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe45c0a8e40 0x7fe45c139b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 1 -- 172.21.15.69:0/590401767 >> 172.21.15.69:0/590401767 conn(0x7fe45c0b7180 msgr2=0x7fe45c0b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:10.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 1 -- 172.21.15.69:0/590401767 shutdown_connections 2023-10-30T20:24:10.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 1 -- 172.21.15.69:0/590401767 wait complete. 2023-10-30T20:24:10.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.531+0000 7fe450ff9640 1 librados: shutdown 2023-10-30T20:24:10.548 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.6 2023-10-30T20:24:10.548 DEBUG:teuthology.parallel:result is None 2023-10-30T20:24:10.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.547+0000 7f4a49c83640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T20:24:10.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.547+0000 7f4a49c83640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T20:24:10.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.547+0000 7f4a49c83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:10.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.547+0000 7f4a49c83640 1 -- 172.21.15.69:0/1337279337 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f4a440abd00 con 0x7f4a44143930 2023-10-30T20:24:10.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.556+0000 7f4a2bfff640 1 -- 172.21.15.69:0/1337279337 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4a38057f80 con 0x7f4a44143930 2023-10-30T20:24:10.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.556+0000 7f4a2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T20:24:10.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.556+0000 7f4a2bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:10.560 INFO:teuthology.orchestra.run.smithi069.stdout:60129542150 2023-10-30T20:24:10.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.558+0000 7f4a49c83640 10 librados: watch_flush enter 2023-10-30T20:24:10.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.558+0000 7f4a49c83640 10 librados: watch_flush exit 2023-10-30T20:24:10.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.558+0000 7f4a49c83640 20 client.4341.objecter shutdown clearing up homeless session... 2023-10-30T20:24:10.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.558+0000 7f4a49c83640 10 client.4341.objecter successfully canceled tick 2023-10-30T20:24:10.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.558+0000 7f4a49c83640 1 -- 172.21.15.69:0/1337279337 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4a3006c7c0 msgr2=0x7f4a3006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.558+0000 7f4a49c83640 1 --2- 172.21.15.69:0/1337279337 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4a3006c7c0 0x7f4a3006ec80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4a34034000 tx=0x7f4a34008030 comp rx=0 tx=0).stop 2023-10-30T20:24:10.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.558+0000 7f4a49c83640 10 monclient: shutdown 2023-10-30T20:24:10.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.558+0000 7f4a49c83640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:10.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.558+0000 7f4a49c83640 1 -- 172.21.15.69:0/1337279337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a44143930 msgr2=0x7f4a441445c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.558+0000 7f4a49c83640 1 --2- 172.21.15.69:0/1337279337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a44143930 0x7f4a441445c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800c7c0 tx=0x7f4a3800cc80 comp rx=0 tx=0).stop 2023-10-30T20:24:10.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.559+0000 7f4a49c83640 1 -- 172.21.15.69:0/1337279337 shutdown_connections 2023-10-30T20:24:10.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.559+0000 7f4a49c83640 1 --2- 172.21.15.69:0/1337279337 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4a3006c7c0 0x7f4a3006ec80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.564 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.559+0000 7f4a49c83640 1 --2- 172.21.15.69:0/1337279337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a441d6090 0x7f4a441d6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.564 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.559+0000 7f4a49c83640 1 --2- 172.21.15.69:0/1337279337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a44143930 0x7f4a441445c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.564 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.559+0000 7f4a49c83640 1 --2- 172.21.15.69:0/1337279337 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c003970 0x7f4a44144080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.564 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.559+0000 7f4a49c83640 1 -- 172.21.15.69:0/1337279337 >> 172.21.15.69:0/1337279337 conn(0x7f4a440b7180 msgr2=0x7f4a4413ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:10.564 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.559+0000 7f4a49c83640 1 -- 172.21.15.69:0/1337279337 shutdown_connections 2023-10-30T20:24:10.564 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.559+0000 7f4a49c83640 1 -- 172.21.15.69:0/1337279337 wait complete. 2023-10-30T20:24:10.565 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.559+0000 7f4a49c83640 1 librados: shutdown 2023-10-30T20:24:10.572 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.2 2023-10-30T20:24:10.572 DEBUG:teuthology.parallel:result is None 2023-10-30T20:24:10.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.703+0000 7f530494e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T20:24:10.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.703+0000 7f530494e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T20:24:10.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.703+0000 7f530494e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:24:10.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.703+0000 7f530494e640 1 -- 172.21.15.69:0/1562146012 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f5300143d80 con 0x7f53001425c0 2023-10-30T20:24:10.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.709+0000 7f52eeffd640 1 -- 172.21.15.69:0/1562146012 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f52f4057d90 con 0x7f53001425c0 2023-10-30T20:24:10.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.709+0000 7f52eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T20:24:10.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.709+0000 7f52eeffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:10.712 INFO:teuthology.orchestra.run.smithi069.stdout:68719476743 2023-10-30T20:24:10.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 10 librados: watch_flush enter 2023-10-30T20:24:10.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 10 librados: watch_flush exit 2023-10-30T20:24:10.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 20 client.4363.objecter shutdown clearing up homeless session... 2023-10-30T20:24:10.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 10 client.4363.objecter successfully canceled tick 2023-10-30T20:24:10.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 1 -- 172.21.15.69:0/1562146012 >> v2:172.21.15.69:6800/1876307353 conn(0x7f52e006c7c0 msgr2=0x7f52e006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 1 --2- 172.21.15.69:0/1562146012 >> v2:172.21.15.69:6800/1876307353 conn(0x7f52e006c7c0 0x7f52e006ec80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f52f0005880 tx=0x7f52f0001150 comp rx=0 tx=0).stop 2023-10-30T20:24:10.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 10 monclient: shutdown 2023-10-30T20:24:10.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:10.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 1 -- 172.21.15.69:0/1562146012 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53001425c0 msgr2=0x7f530013b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 1 --2- 172.21.15.69:0/1562146012 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53001425c0 0x7f530013b1b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f52f400c440 tx=0x7f52f4001e70 comp rx=0 tx=0).stop 2023-10-30T20:24:10.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 1 -- 172.21.15.69:0/1562146012 shutdown_connections 2023-10-30T20:24:10.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 1 --2- 172.21.15.69:0/1562146012 >> v2:172.21.15.69:6800/1876307353 conn(0x7f52e006c7c0 0x7f52e006ec80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 1 --2- 172.21.15.69:0/1562146012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53001d7680 0x7f530013b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 1 --2- 172.21.15.69:0/1562146012 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53001425c0 0x7f530013b1b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 1 --2- 172.21.15.69:0/1562146012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53000a8e40 0x7f53001d6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 1 -- 172.21.15.69:0/1562146012 >> 172.21.15.69:0/1562146012 conn(0x7f53000b7180 msgr2=0x7f53000afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:10.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 1 -- 172.21.15.69:0/1562146012 shutdown_connections 2023-10-30T20:24:10.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 1 -- 172.21.15.69:0/1562146012 wait complete. 2023-10-30T20:24:10.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.711+0000 7f530494e640 1 librados: shutdown 2023-10-30T20:24:10.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.718+0000 7f0b7ccb4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T20:24:10.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.718+0000 7f0b7ccb4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T20:24:10.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.718+0000 7f0b7ccb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:10.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.718+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/2023654174 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0b780afd50 con 0x7f0b78144f50 2023-10-30T20:24:10.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.722+0000 7f0b66ffd640 1 -- 172.21.15.69:0/2023654174 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0b70060b70 con 0x7f0b78144f50 2023-10-30T20:24:10.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.722+0000 7f0b66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T20:24:10.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.722+0000 7f0b66ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:10.724 INFO:teuthology.orchestra.run.smithi069.stdout:64424509444 2023-10-30T20:24:10.725 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476743 for osd.0 2023-10-30T20:24:10.726 DEBUG:teuthology.parallel:result is None 2023-10-30T20:24:10.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.723+0000 7f0b7ccb4640 10 librados: watch_flush enter 2023-10-30T20:24:10.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.723+0000 7f0b7ccb4640 10 librados: watch_flush exit 2023-10-30T20:24:10.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.723+0000 7f0b7ccb4640 20 client.4322.objecter shutdown clearing up homeless session... 2023-10-30T20:24:10.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.723+0000 7f0b7ccb4640 10 client.4322.objecter successfully canceled tick 2023-10-30T20:24:10.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.723+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/2023654174 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0b5806c7f0 msgr2=0x7f0b5806ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.723+0000 7f0b7ccb4640 1 --2- 172.21.15.69:0/2023654174 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0b5806c7f0 0x7f0b5806ecb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0b68035000 tx=0x7f0b68030000 comp rx=0 tx=0).stop 2023-10-30T20:24:10.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.723+0000 7f0b7ccb4640 10 monclient: shutdown 2023-10-30T20:24:10.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.723+0000 7f0b7ccb4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:10.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.723+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/2023654174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b78144f50 msgr2=0x7f0b781da310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.723+0000 7f0b7ccb4640 1 --2- 172.21.15.69:0/2023654174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b78144f50 0x7f0b781da310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0b7000b970 tx=0x7f0b7000be30 comp rx=0 tx=0).stop 2023-10-30T20:24:10.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.724+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/2023654174 shutdown_connections 2023-10-30T20:24:10.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.724+0000 7f0b7ccb4640 1 --2- 172.21.15.69:0/2023654174 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0b5806c7f0 0x7f0b5806ecb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.724+0000 7f0b7ccb4640 1 --2- 172.21.15.69:0/2023654174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b78144f50 0x7f0b781da310 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.724+0000 7f0b7ccb4640 1 --2- 172.21.15.69:0/2023654174 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7813eb50 0x7f0b7813fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.724+0000 7f0b7ccb4640 1 --2- 172.21.15.69:0/2023654174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b780b4350 0x7f0b780b0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.724+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/2023654174 >> 172.21.15.69:0/2023654174 conn(0x7f0b78065ea0 msgr2=0x7f0b7813c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:10.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.724+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/2023654174 shutdown_connections 2023-10-30T20:24:10.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.724+0000 7f0b7ccb4640 1 -- 172.21.15.69:0/2023654174 wait complete. 2023-10-30T20:24:10.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.724+0000 7f0b7ccb4640 1 librados: shutdown 2023-10-30T20:24:10.737 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.5 2023-10-30T20:24:10.737 DEBUG:teuthology.parallel:result is None 2023-10-30T20:24:10.737 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T20:24:10.738 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T20:24:10.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.874+0000 7f43d56d7640 10 monclient: get_monmap_and_config 2023-10-30T20:24:10.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.874+0000 7f43d56d7640 10 monclient: build_initial_monmap 2023-10-30T20:24:10.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.874+0000 7f43d56d7640 10 monclient: monmap: 2023-10-30T20:24:10.876 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:10.877 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.877 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:10.875812+0000 2023-10-30T20:24:10.877 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:10.875812+0000 2023-10-30T20:24:10.877 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:10.877 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.878 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:10.878 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:10.878 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:10.878 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:10.878 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:10.879 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.874+0000 7f43d56d7640 1 Processor -- start 2023-10-30T20:24:10.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.874+0000 7f43d56d7640 1 -- start start 2023-10-30T20:24:10.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.874+0000 7f43d56d7640 10 monclient: init 2023-10-30T20:24:10.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43d56d7640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:10.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43d56d7640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-10-30T20:24:10.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43d56d7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f43d00ad660 0x7f43d006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43d56d7640 10 monclient(hunting): picked mon.noname-a con 0x7f43d00ad660 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:10.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43d56d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d00ada40 0x7f43d0141af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43d56d7640 10 monclient(hunting): picked mon.noname-b con 0x7f43d00ada40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:10.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43d56d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43d013bb30 addr v1:172.21.15.69:6789/0 2023-10-30T20:24:10.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43d56d7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d0070cb0 con 0x7f43d013bb30 2023-10-30T20:24:10.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43d56d7640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:10.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43d56d7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d0067b90 con 0x7f43d00ad660 2023-10-30T20:24:10.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43d56d7640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:10.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43d56d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d0071ef0 con 0x7f43d00ada40 2023-10-30T20:24:10.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43d56d7640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:10.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43d56d7640 10 monclient(hunting): authenticate will time out at 1574.164914s 2023-10-30T20:24:10.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43ceffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f43d00ad660 0x7f43d006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.875+0000 7f43ce7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d00ada40 0x7f43d0141af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.876+0000 7f43ce7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d00ada40 0x7f43d0141af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:40704/0 (socket says 172.21.15.69:40704) 2023-10-30T20:24:10.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.876+0000 7f43ce7fc640 1 -- 172.21.15.69:0/2249113570 learned_addr learned my addr 172.21.15.69:0/2249113570 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:10.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.876+0000 7f43ce7fc640 10 monclient(hunting): get_auth_request con 0x7f43d00ada40 auth_method 0 2023-10-30T20:24:10.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.876+0000 7f43ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:10.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.876+0000 7f43ce7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:10.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.876+0000 7f43ce7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:10.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.876+0000 7f43ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:10.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.876+0000 7f43ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:10.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.876+0000 7f43ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:10.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.876+0000 7f43ceffd640 10 monclient(hunting): get_auth_request con 0x7f43d00ad660 auth_method 0 2023-10-30T20:24:10.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.876+0000 7f43ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:10.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.876+0000 7f43ceffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:10.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.876+0000 7f43ceffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:10.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.877+0000 7f43ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:10.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.877+0000 7f43ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:10.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.877+0000 7f43ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:10.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.878+0000 7f43ce7fc640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2023-10-30T20:24:10.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.878+0000 7f43ce7fc640 1 -- 172.21.15.69:0/2249113570 >> v2:172.21.15.69:3300/0 conn(0x7f43d00ad660 msgr2=0x7f43d006ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:10.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.878+0000 7f43ce7fc640 1 --2- 172.21.15.69:0/2249113570 >> v2:172.21.15.69:3300/0 conn(0x7f43d00ad660 0x7f43d006ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.878+0000 7f43ce7fc640 1 -- 172.21.15.69:0/2249113570 >> v1:172.21.15.69:6789/0 conn(0x7f43d013bb30 legacy=0x7f43d00717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.878+0000 7f43ce7fc640 10 monclient: _finish_hunting 0 2023-10-30T20:24:10.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.878+0000 7f43ce7fc640 1 monclient: found mon.noname-b 2023-10-30T20:24:10.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.878+0000 7f43ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:10.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.878+0000 7f43ce7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:24:10.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.878+0000 7f43ce7fc640 1 -- 172.21.15.69:0/2249113570 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d0071300 con 0x7f43d00ada40 2023-10-30T20:24:10.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.880+0000 7f43ce7fc640 1 --2- 172.21.15.69:0/2249113570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d00ada40 0x7f43d0141af0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f43c400bb10 tx=0x7f43c402e0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81469fac02217bde server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:10.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.883+0000 7f43cdffb640 1 -- 172.21.15.69:0/2249113570 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c402ee70 con 0x7f43d00ada40 2023-10-30T20:24:10.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:10.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43cdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:10.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43cdffb640 10 monclient: dump: 2023-10-30T20:24:10.894 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:10.894 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.894 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.895 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.895 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:10.895 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.895 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:10.895 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:10.896 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:10.896 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43cdffb640 10 monclient: _finish_auth 0 2023-10-30T20:24:10.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43cdffb640 10 monclient: _check_auth_tickets 2023-10-30T20:24:10.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:10.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43cdffb640 1 -- 172.21.15.69:0/2249113570 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c4036440 con 0x7f43d00ada40 2023-10-30T20:24:10.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43cdffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:10.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43cdffb640 1 -- 172.21.15.69:0/2249113570 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c40367c0 con 0x7f43d00ada40 2023-10-30T20:24:10.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 5 monclient: authenticate success, global_id 4325 2023-10-30T20:24:10.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:10.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 4 set_mon_vals no callback set 2023-10-30T20:24:10.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:10.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:10.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43cdffb640 10 monclient: dump: 2023-10-30T20:24:10.898 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:10.898 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.899 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.899 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.899 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:10.899 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.899 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:10.899 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:10.900 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:10.900 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 10 monclient: shutdown 2023-10-30T20:24:10.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:10.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 1 -- 172.21.15.69:0/2249113570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d00ada40 msgr2=0x7f43d0141af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 1 --2- 172.21.15.69:0/2249113570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d00ada40 0x7f43d0141af0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f43c400bb10 tx=0x7f43c402e0c0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 1 -- 172.21.15.69:0/2249113570 shutdown_connections 2023-10-30T20:24:10.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 1 --2- 172.21.15.69:0/2249113570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d00ada40 0x7f43d0141af0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 1 --2- 172.21.15.69:0/2249113570 >> v2:172.21.15.69:3300/0 conn(0x7f43d00ad660 0x7f43d006ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 1 -- 172.21.15.69:0/2249113570 >> 172.21.15.69:0/2249113570 conn(0x7f43d00afb80 msgr2=0x7f43d00aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:10.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 1 -- 172.21.15.69:0/2249113570 shutdown_connections 2023-10-30T20:24:10.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 1 -- 172.21.15.69:0/2249113570 wait complete. 2023-10-30T20:24:10.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 10 monclient: build_initial_monmap 2023-10-30T20:24:10.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 10 monclient: monmap: 2023-10-30T20:24:10.902 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:10.902 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:10.903 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:10.903 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:10.903 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:10.903 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:10.903 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:10.904 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 1 librados: starting msgr at 2023-10-30T20:24:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.885+0000 7f43d56d7640 1 librados: starting objecter 2023-10-30T20:24:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 1 Processor -- start 2023-10-30T20:24:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 1 -- start start 2023-10-30T20:24:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 1 librados: setting wanted keys 2023-10-30T20:24:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 1 librados: calling monclient init 2023-10-30T20:24:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 10 monclient: init 2023-10-30T20:24:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T20:24:10.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d00ad660 0x7f43d0140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 10 monclient(hunting): picked mon.noname-c con 0x7f43d00ad660 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:10.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43d013bb30 0x7f43d01d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 10 monclient(hunting): picked mon.noname-a con 0x7f43d013bb30 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:10.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43d01d2de0 0x7f43d00ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 10 monclient(hunting): picked mon.noname-b con 0x7f43d01d2de0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:10.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:10.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d006e2b0 con 0x7f43d013bb30 2023-10-30T20:24:10.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:10.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d013c3d0 con 0x7f43d01d2de0 2023-10-30T20:24:10.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:10.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d00abc00 con 0x7f43d00ad660 2023-10-30T20:24:10.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:10.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.886+0000 7f43d56d7640 10 monclient(hunting): authenticate will time out at 1574.176570s 2023-10-30T20:24:10.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.887+0000 7f43ceffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d00ad660 0x7f43d0140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.887+0000 7f43ce7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43d013bb30 0x7f43d01d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.887+0000 7f43cf7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43d01d2de0 0x7f43d00ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.887+0000 7f43ceffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d00ad660 0x7f43d0140f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:40718/0 (socket says 172.21.15.69:40718) 2023-10-30T20:24:10.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.887+0000 7f43ceffd640 1 -- 172.21.15.69:0/289044659 learned_addr learned my addr 172.21.15.69:0/289044659 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:10.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.887+0000 7f43ceffd640 10 monclient(hunting): get_auth_request con 0x7f43d00ad660 auth_method 0 2023-10-30T20:24:10.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.887+0000 7f43ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:10.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.887+0000 7f43ceffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:10.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.887+0000 7f43ceffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:10.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.887+0000 7f43ce7fc640 10 monclient(hunting): get_auth_request con 0x7f43d013bb30 auth_method 0 2023-10-30T20:24:10.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.887+0000 7f43ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:10.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.887+0000 7f43ce7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:10.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.887+0000 7f43ce7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:10.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.888+0000 7f43cf7fe640 10 monclient(hunting): get_auth_request con 0x7f43d01d2de0 auth_method 0 2023-10-30T20:24:10.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.888+0000 7f43cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:10.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.888+0000 7f43cf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:10.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.888+0000 7f43cf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:10.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.888+0000 7f43ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:10.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.888+0000 7f43ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:10.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.888+0000 7f43ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:10.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.888+0000 7f43ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:10.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.888+0000 7f43ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:10.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.888+0000 7f43ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:10.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.888+0000 7f43cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:10.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.888+0000 7f43cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:10.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.888+0000 7f43cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:10.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.891+0000 7f43ceffd640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2023-10-30T20:24:10.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.891+0000 7f43ceffd640 1 -- 172.21.15.69:0/289044659 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43d01d2de0 msgr2=0x7f43d00ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.891+0000 7f43ceffd640 1 --2- 172.21.15.69:0/289044659 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43d01d2de0 0x7f43d00ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.891+0000 7f43ceffd640 1 -- 172.21.15.69:0/289044659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43d013bb30 msgr2=0x7f43d01d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:10.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.891+0000 7f43ceffd640 1 --2- 172.21.15.69:0/289044659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43d013bb30 0x7f43d01d28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:10.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.891+0000 7f43ceffd640 10 monclient: _finish_hunting 0 2023-10-30T20:24:10.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.891+0000 7f43ceffd640 1 monclient: found mon.noname-c 2023-10-30T20:24:10.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.891+0000 7f43ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:10.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.891+0000 7f43ceffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:24:10.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.891+0000 7f43ceffd640 1 -- 172.21.15.69:0/289044659 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d00abdd0 con 0x7f43d00ad660 2023-10-30T20:24:10.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.891+0000 7f43ce7fc640 1 --2- 172.21.15.69:0/289044659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43d013bb30 0x7f43d01d28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T20:24:10.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.893+0000 7f43ceffd640 1 --2- 172.21.15.69:0/289044659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d00ad660 0x7f43d0140f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f43c000ea10 tx=0x7f43c000eed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:10.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.893+0000 7f43b77fe640 10 client.?.objecter ms_handle_connect 0x7f43d00ad660 2023-10-30T20:24:10.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.893+0000 7f43b77fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:10.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.896+0000 7f43b77fe640 1 -- 172.21.15.69:0/289044659 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c000c9a0 con 0x7f43d00ad660 2023-10-30T20:24:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:24:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43b77fe640 10 monclient: dump: 2023-10-30T20:24:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.918 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.918 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.918 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:10.918 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.918 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:10.919 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:10.919 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:10.919 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43b77fe640 10 monclient: _finish_auth 0 2023-10-30T20:24:10.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43b77fe640 10 monclient: _check_auth_tickets 2023-10-30T20:24:10.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:10.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43b77fe640 1 -- 172.21.15.69:0/289044659 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c000d040 con 0x7f43d00ad660 2023-10-30T20:24:10.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43b77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:10.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43b77fe640 1 -- 172.21.15.69:0/289044659 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c000cd50 con 0x7f43d00ad660 2023-10-30T20:24:10.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43d56d7640 5 monclient: authenticate success, global_id 4328 2023-10-30T20:24:10.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43d56d7640 10 monclient: _renew_subs 2023-10-30T20:24:10.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43d56d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:10.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43d56d7640 1 -- 172.21.15.69:0/289044659 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d00ac090 con 0x7f43d00ad660 2023-10-30T20:24:10.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:10.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:10.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43cd7fa640 4 set_mon_vals no callback set 2023-10-30T20:24:10.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43b77fe640 10 monclient: dump: 2023-10-30T20:24:10.922 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:10.922 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:10.922 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.923 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:10.923 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:10.923 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:10.923 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:10.923 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:10.924 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43d56d7640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43d56d7640 10 monclient: _renew_subs 2023-10-30T20:24:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43d56d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:10.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43d56d7640 1 -- 172.21.15.69:0/289044659 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d013ca40 con 0x7f43d00ad660 2023-10-30T20:24:10.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.897+0000 7f43d56d7640 1 librados: init done 2023-10-30T20:24:10.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.898+0000 7f43d56d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:10.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.898+0000 7f43d56d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:10.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.898+0000 7f43d56d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:24:10.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.898+0000 7f43d56d7640 1 -- 172.21.15.69:0/289044659 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d004e9b0 con 0x7f43d00ad660 2023-10-30T20:24:10.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.915+0000 7f43b77fe640 1 -- 172.21.15.69:0/289044659 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f43c0013470 con 0x7f43d00ad660 2023-10-30T20:24:10.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.915+0000 7f43b77fe640 1 --2- 172.21.15.69:0/289044659 >> v2:172.21.15.69:6800/1876307353 conn(0x7f43bc06c810 0x7f43bc06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:10.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.915+0000 7f43ce7fc640 1 --2- 172.21.15.69:0/289044659 >> v2:172.21.15.69:6800/1876307353 conn(0x7f43bc06c810 0x7f43bc06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:10.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.915+0000 7f43ce7fc640 10 monclient: get_auth_request con 0x7f43bc06c810 auth_method 0 2023-10-30T20:24:10.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.915+0000 7f43ce7fc640 1 --2- 172.21.15.69:0/289044659 >> v2:172.21.15.69:6800/1876307353 conn(0x7f43bc06c810 0x7f43bc06ecd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f43c4038000 tx=0x7f43c4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:10.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.915+0000 7f43b77fe640 10 client.4328.objecter ms_handle_connect 0x7f43bc06c810 2023-10-30T20:24:10.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.917+0000 7f43b77fe640 1 -- 172.21.15.69:0/289044659 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f43c00586e0 con 0x7f43d00ad660 2023-10-30T20:24:10.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.917+0000 7f43b77fe640 10 client.4328.objecter ms_dispatch 0x7f43d0000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:10.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.917+0000 7f43b77fe640 3 client.4328.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:10.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.917+0000 7f43b77fe640 3 client.4328.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:10.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:10.917+0000 7f43b77fe640 20 client.4328.objecter dump_active .. 0 homeless 2023-10-30T20:24:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.149+0000 7f43b77fe640 1 -- 172.21.15.69:0/289044659 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f43c005c780 con 0x7f43d00ad660 2023-10-30T20:24:11.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.149+0000 7f43b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:11.152 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.149+0000 7f43b77fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:11.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.312+0000 7f43d56d7640 1 -- 172.21.15.69:0/289044659 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f43d013c520 con 0x7f43bc06c810 2023-10-30T20:24:11.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.315+0000 7f43b77fe640 1 -- 172.21.15.69:0/289044659 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48061 (secure 0 0 0) 0x7f43d013c520 con 0x7f43bc06c810 2023-10-30T20:24:11.318 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:24:11.318 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-10-30T20:24:11.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 10 librados: watch_flush enter 2023-10-30T20:24:11.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 10 librados: watch_flush exit 2023-10-30T20:24:11.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 20 client.4328.objecter shutdown clearing up homeless session... 2023-10-30T20:24:11.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 10 client.4328.objecter successfully canceled tick 2023-10-30T20:24:11.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 1 -- 172.21.15.69:0/289044659 >> v2:172.21.15.69:6800/1876307353 conn(0x7f43bc06c810 msgr2=0x7f43bc06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:11.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 1 --2- 172.21.15.69:0/289044659 >> v2:172.21.15.69:6800/1876307353 conn(0x7f43bc06c810 0x7f43bc06ecd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f43c4038000 tx=0x7f43c4032000 comp rx=0 tx=0).stop 2023-10-30T20:24:11.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 10 monclient: shutdown 2023-10-30T20:24:11.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:11.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 1 -- 172.21.15.69:0/289044659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d00ad660 msgr2=0x7f43d0140f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:11.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 1 --2- 172.21.15.69:0/289044659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d00ad660 0x7f43d0140f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f43c000ea10 tx=0x7f43c000eed0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 1 -- 172.21.15.69:0/289044659 shutdown_connections 2023-10-30T20:24:11.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 1 --2- 172.21.15.69:0/289044659 >> v2:172.21.15.69:6800/1876307353 conn(0x7f43bc06c810 0x7f43bc06ecd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 1 --2- 172.21.15.69:0/289044659 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43d01d2de0 0x7f43d00ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 1 --2- 172.21.15.69:0/289044659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43d013bb30 0x7f43d01d28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 1 --2- 172.21.15.69:0/289044659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d00ad660 0x7f43d0140f30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 1 -- 172.21.15.69:0/289044659 >> 172.21.15.69:0/289044659 conn(0x7f43d00afb80 msgr2=0x7f43d0139c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:11.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 1 -- 172.21.15.69:0/289044659 shutdown_connections 2023-10-30T20:24:11.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 1 -- 172.21.15.69:0/289044659 wait complete. 2023-10-30T20:24:11.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.317+0000 7f43d56d7640 1 librados: shutdown 2023-10-30T20:24:11.330 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-30T20:24:10.172102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217700,"kb_used_data":1496,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757020,"statfs":{"total":773094113280,"available":772871188480,"internally_reserved":0,"allocated":1531904,"data_stored":1281464,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12745,"internal_metadata":221236791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.252186"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T20:24:06.843187+0000","last_change":"2023-10-30T20:24:04.700492+0000","last_active":"2023-10-30T20:24:06.843187+0000","last_peered":"2023-10-30T20:24:06.843187+0000","last_clean":"2023-10-30T20:24:06.843187+0000","last_became_active":"2023-10-30T20:24:04.692769+0000","last_became_peered":"2023-10-30T20:24:04.692769+0000","last_unstale":"2023-10-30T20:24:06.843187+0000","last_undegraded":"2023-10-30T20:24:06.843187+0000","last_fullsized":"2023-10-30T20:24:06.843187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:58.321494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T20:24:05.227061+0000","last_change":"2023-10-30T20:24:04.738314+0000","last_active":"2023-10-30T20:24:05.227061+0000","last_peered":"2023-10-30T20:24:05.227061+0000","last_clean":"2023-10-30T20:24:05.227061+0000","last_became_active":"2023-10-30T20:24:04.712006+0000","last_became_peered":"2023-10-30T20:24:04.712006+0000","last_unstale":"2023-10-30T20:24:05.227061+0000","last_undegraded":"2023-10-30T20:24:05.227061+0000","last_fullsized":"2023-10-30T20:24:05.227061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:41:08.136073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T20:24:06.848831+0000","last_change":"2023-10-30T20:24:04.315228+0000","last_active":"2023-10-30T20:24:06.848831+0000","last_peered":"2023-10-30T20:24:06.848831+0000","last_clean":"2023-10-30T20:24:06.848831+0000","last_became_active":"2023-10-30T20:24:04.310065+0000","last_became_peered":"2023-10-30T20:24:04.310065+0000","last_unstale":"2023-10-30T20:24:06.848831+0000","last_undegraded":"2023-10-30T20:24:06.848831+0000","last_fullsized":"2023-10-30T20:24:06.848831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:09:06.876832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T20:24:08.354070+0000","last_change":"2023-10-30T20:24:04.552803+0000","last_active":"2023-10-30T20:24:08.354070+0000","last_peered":"2023-10-30T20:24:08.354070+0000","last_clean":"2023-10-30T20:24:08.354070+0000","last_became_active":"2023-10-30T20:24:04.525673+0000","last_became_peered":"2023-10-30T20:24:04.525673+0000","last_unstale":"2023-10-30T20:24:08.354070+0000","last_undegraded":"2023-10-30T20:24:08.354070+0000","last_fullsized":"2023-10-30T20:24:08.354070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:18.627320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T20:24:06.822671+0000","last_change":"2023-10-30T20:24:04.525394+0000","last_active":"2023-10-30T20:24:06.822671+0000","last_peered":"2023-10-30T20:24:06.822671+0000","last_clean":"2023-10-30T20:24:06.822671+0000","last_became_active":"2023-10-30T20:24:04.512341+0000","last_became_peered":"2023-10-30T20:24:04.512341+0000","last_unstale":"2023-10-30T20:24:06.822671+0000","last_undegraded":"2023-10-30T20:24:06.822671+0000","last_fullsized":"2023-10-30T20:24:06.822671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:00:36.886515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T20:24:06.765719+0000","last_change":"2023-10-30T20:24:05.504482+0000","last_active":"2023-10-30T20:24:06.765719+0000","last_peered":"2023-10-30T20:24:06.765719+0000","last_clean":"2023-10-30T20:24:06.765719+0000","last_became_active":"2023-10-30T20:24:05.475394+0000","last_became_peered":"2023-10-30T20:24:05.475394+0000","last_unstale":"2023-10-30T20:24:06.765719+0000","last_undegraded":"2023-10-30T20:24:06.765719+0000","last_fullsized":"2023-10-30T20:24:06.765719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:21:02.335096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T20:24:06.810235+0000","last_change":"2023-10-30T20:24:04.284202+0000","last_active":"2023-10-30T20:24:06.810235+0000","last_peered":"2023-10-30T20:24:06.810235+0000","last_clean":"2023-10-30T20:24:06.810235+0000","last_became_active":"2023-10-30T20:24:04.254051+0000","last_became_peered":"2023-10-30T20:24:04.254051+0000","last_unstale":"2023-10-30T20:24:06.810235+0000","last_undegraded":"2023-10-30T20:24:06.810235+0000","last_fullsized":"2023-10-30T20:24:06.810235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:14:06.705421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T20:24:06.827697+0000","last_change":"2023-10-30T20:24:04.676966+0000","last_active":"2023-10-30T20:24:06.827697+0000","last_peered":"2023-10-30T20:24:06.827697+0000","last_clean":"2023-10-30T20:24:06.827697+0000","last_became_active":"2023-10-30T20:24:04.671790+0000","last_became_peered":"2023-10-30T20:24:04.671790+0000","last_unstale":"2023-10-30T20:24:06.827697+0000","last_undegraded":"2023-10-30T20:24:06.827697+0000","last_fullsized":"2023-10-30T20:24:06.827697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:56.600896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":117,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T20:24:06.245375+0000","last_change":"2023-10-30T20:23:57.127569+0000","last_active":"2023-10-30T20:24:06.245375+0000","last_peered":"2023-10-30T20:24:06.245375+0000","last_clean":"2023-10-30T20:24:06.245375+0000","last_became_active":"2023-10-30T20:23:57.098872+0000","last_became_peered":"2023-10-30T20:23:57.098872+0000","last_unstale":"2023-10-30T20:24:06.245375+0000","last_undegraded":"2023-10-30T20:24:06.245375+0000","last_fullsized":"2023-10-30T20:24:06.245375+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:23:54.534733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:23:54.534733+0000","last_clean_scrub_stamp":"2023-10-30T20:23:54.534733+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:26:29.039781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476742,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":59566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.202999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.4}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.534000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5860000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.058999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.420000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8529999999999998}]}]},{"osd":1,"up_from":16,"seq":68719476741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":457406,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9380000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.821000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0809999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.962}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.052}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.959}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.699}]}]},{"osd":0,"up_from":16,"seq":68719476743,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":59566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.934000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.865}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1370000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.205}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.896000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.763999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.702999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":64194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7360000000000007}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.309000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.368}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.782}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.501000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.173999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.126000000000001}]}]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":457406,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.113}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.249000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.219000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.042999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.590999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.012}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.459}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":188416,"data_stored":64194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.0720000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.515000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.59}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.314}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.420999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.542000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.56}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":59566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8019999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4339999999999993}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.241}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.031000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.128}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.343}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":34.646000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":59566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.148}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.48}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.056000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.321000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.673}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":177.608}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.788}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T20:24:11.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T20:24:11.436 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.434+0000 7f4844494640 10 monclient: get_monmap_and_config 2023-10-30T20:24:11.436 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.434+0000 7f4844494640 10 monclient: build_initial_monmap 2023-10-30T20:24:11.436 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.434+0000 7f4844494640 10 monclient: monmap: 2023-10-30T20:24:11.436 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:11.437 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:11.437 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:11.435767+0000 2023-10-30T20:24:11.437 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:11.435767+0000 2023-10-30T20:24:11.437 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:11.437 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:11.438 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:11.438 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:11.438 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:11.438 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:11.438 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:11.439 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:11.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.435+0000 7f4844494640 1 Processor -- start 2023-10-30T20:24:11.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.435+0000 7f4844494640 1 -- start start 2023-10-30T20:24:11.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.435+0000 7f4844494640 10 monclient: init 2023-10-30T20:24:11.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.436+0000 7f4844494640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:11.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.436+0000 7f4844494640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-10-30T20:24:11.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.436+0000 7f4844494640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f483c0b4390 0x7f483c0a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:11.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.436+0000 7f4844494640 10 monclient(hunting): picked mon.noname-b con 0x7f483c0b4390 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:11.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.436+0000 7f4844494640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f483c1418a0 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:11.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.436+0000 7f4844494640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f483c141ce0 0x7f483c0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:11.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.436+0000 7f4844494640 10 monclient(hunting): picked mon.noname-a con 0x7f483c141ce0 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:11.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.436+0000 7f4844494640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c0a8f60 con 0x7f483c1418a0 2023-10-30T20:24:11.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.436+0000 7f4844494640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:11.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.436+0000 7f4844494640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483c067a10 con 0x7f483c141ce0 2023-10-30T20:24:11.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.436+0000 7f4844494640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:11.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.436+0000 7f4844494640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c0b4770 con 0x7f483c0b4390 2023-10-30T20:24:11.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.436+0000 7f4844494640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:11.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.436+0000 7f4844494640 10 monclient(hunting): authenticate will time out at 1574.725121s 2023-10-30T20:24:11.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.438+0000 7f4842230640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f483c0b4390 0x7f483c0a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:11.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.439+0000 7f4841a2f640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f483c1418a0 0x7f483c0b1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.69:51882/0 (socket says 172.21.15.69:51882) 2023-10-30T20:24:11.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.439+0000 7f4841a2f640 1 -- 172.21.15.69:0/2849574922 learned_addr learned my addr 172.21.15.69:0/2849574922 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:24:11.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.439+0000 7f4842230640 10 monclient(hunting): get_auth_request con 0x7f483c0b4390 auth_method 0 2023-10-30T20:24:11.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.439+0000 7f4842230640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:11.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.439+0000 7f4842230640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:11.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.439+0000 7f4842230640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:11.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.440+0000 7f4842230640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:11.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.440+0000 7f4842230640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:11.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.440+0000 7f4842230640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:11.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.442+0000 7f4842230640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2023-10-30T20:24:11.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.442+0000 7f4842230640 1 -- 172.21.15.69:0/2849574922 >> v2:172.21.15.69:3300/0 conn(0x7f483c141ce0 msgr2=0x7f483c0b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:11.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.442+0000 7f4842230640 1 --2- 172.21.15.69:0/2849574922 >> v2:172.21.15.69:3300/0 conn(0x7f483c141ce0 0x7f483c0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.442+0000 7f4842230640 1 -- 172.21.15.69:0/2849574922 >> v1:172.21.15.183:6789/0 conn(0x7f483c1418a0 legacy=0x7f483c0b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:11.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.442+0000 7f4842230640 10 monclient: _finish_hunting 0 2023-10-30T20:24:11.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.442+0000 7f4842230640 1 monclient: found mon.noname-b 2023-10-30T20:24:11.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.442+0000 7f4842230640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:11.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.442+0000 7f4842230640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:24:11.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.442+0000 7f4842230640 1 -- 172.21.15.69:0/2849574922 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c142170 con 0x7f483c0b4390 2023-10-30T20:24:11.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.444+0000 7f4842230640 1 --2- 172.21.15.69:0/2849574922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f483c0b4390 0x7f483c0a8770 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f482c0097a0 tx=0x7f482c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e387c5943f73607 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:11.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.506+0000 7f484122e640 1 -- 172.21.15.69:0/2849574922 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c031830 con 0x7f483c0b4390 2023-10-30T20:24:11.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.506+0000 7f484122e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.506+0000 7f484122e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:24:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.506+0000 7f484122e640 10 monclient: dump: 2023-10-30T20:24:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:11.510 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:11.510 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:11.510 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:11.510 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:11.510 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:11.511 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:11.511 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:11.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f484122e640 10 monclient: _finish_auth 0 2023-10-30T20:24:11.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f484122e640 10 monclient: _check_auth_tickets 2023-10-30T20:24:11.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f484122e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:11.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f484122e640 1 -- 172.21.15.69:0/2849574922 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482c02d040 con 0x7f483c0b4390 2023-10-30T20:24:11.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f4844494640 5 monclient: authenticate success, global_id 4331 2023-10-30T20:24:11.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f4844494640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:11.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f484122e640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:11.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f4844494640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:11.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f4844494640 4 set_mon_vals no callback set 2023-10-30T20:24:11.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f484122e640 1 -- 172.21.15.69:0/2849574922 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c031bc0 con 0x7f483c0b4390 2023-10-30T20:24:11.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f484122e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:11.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f484122e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:24:11.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f484122e640 10 monclient: dump: 2023-10-30T20:24:11.513 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:11.514 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:11.514 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:11.514 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:11.514 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:11.514 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:11.515 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:11.515 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:11.515 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:11.515 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:11.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f4844494640 10 monclient: shutdown 2023-10-30T20:24:11.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f4844494640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:11.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f4844494640 1 -- 172.21.15.69:0/2849574922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f483c0b4390 msgr2=0x7f483c0a8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:11.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f4844494640 1 --2- 172.21.15.69:0/2849574922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f483c0b4390 0x7f483c0a8770 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f482c0097a0 tx=0x7f482c02b8d0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f4844494640 1 -- 172.21.15.69:0/2849574922 shutdown_connections 2023-10-30T20:24:11.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f4844494640 1 --2- 172.21.15.69:0/2849574922 >> v2:172.21.15.69:3300/0 conn(0x7f483c141ce0 0x7f483c0b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f4844494640 1 --2- 172.21.15.69:0/2849574922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f483c0b4390 0x7f483c0a8770 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f4844494640 1 -- 172.21.15.69:0/2849574922 >> 172.21.15.69:0/2849574922 conn(0x7f483c065ea0 msgr2=0x7f483c0662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:11.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f4844494640 1 -- 172.21.15.69:0/2849574922 shutdown_connections 2023-10-30T20:24:11.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.507+0000 7f4844494640 1 -- 172.21.15.69:0/2849574922 wait complete. 2023-10-30T20:24:11.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.508+0000 7f4844494640 10 monclient: build_initial_monmap 2023-10-30T20:24:11.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.508+0000 7f4844494640 10 monclient: monmap: 2023-10-30T20:24:11.520 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:11.520 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:11.520 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:11.521 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:11.521 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:11.521 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:11.521 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:11.521 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:11.522 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:11.522 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:11.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.508+0000 7f4844494640 1 librados: starting msgr at 2023-10-30T20:24:11.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.508+0000 7f4844494640 1 librados: starting objecter 2023-10-30T20:24:11.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.508+0000 7f4844494640 1 Processor -- start 2023-10-30T20:24:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.508+0000 7f4844494640 1 -- start start 2023-10-30T20:24:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.508+0000 7f4844494640 1 librados: setting wanted keys 2023-10-30T20:24:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.508+0000 7f4844494640 1 librados: calling monclient init 2023-10-30T20:24:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.508+0000 7f4844494640 10 monclient: init 2023-10-30T20:24:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:11.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T20:24:11.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f483c0b4390 0x7f483c1da300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:11.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 10 monclient(hunting): picked mon.noname-b con 0x7f483c0b4390 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:11.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f483c1418a0 0x7f483c0aba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:11.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 10 monclient(hunting): picked mon.noname-a con 0x7f483c1418a0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:11.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f483c141ce0 0x7f483c0abf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:11.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 10 monclient(hunting): picked mon.noname-c con 0x7f483c141ce0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:11.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:11.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c0b12b0 con 0x7f483c1418a0 2023-10-30T20:24:11.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:11.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c0ad850 con 0x7f483c0b4390 2023-10-30T20:24:11.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:11.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c0ab4f0 con 0x7f483c141ce0 2023-10-30T20:24:11.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:11.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4844494640 10 monclient(hunting): authenticate will time out at 1574.798250s 2023-10-30T20:24:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.509+0000 7f4841a2f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f483c1418a0 0x7f483c0aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4842230640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f483c0b4390 0x7f483c1da300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4841a2f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f483c1418a0 0x7f483c0aba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56786/0 (socket says 172.21.15.69:56786) 2023-10-30T20:24:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4841a2f640 1 -- 172.21.15.69:0/1696255916 learned_addr learned my addr 172.21.15.69:0/1696255916 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4841a2f640 10 monclient(hunting): get_auth_request con 0x7f483c1418a0 auth_method 0 2023-10-30T20:24:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4841a2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4841a2f640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4841a2f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4842230640 10 monclient(hunting): get_auth_request con 0x7f483c0b4390 auth_method 0 2023-10-30T20:24:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4842230640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4842230640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:11.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4842230640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:11.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4841a2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:11.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4841a2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:11.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4841a2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:11.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4842230640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:11.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4842230640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:11.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.510+0000 7f4842230640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:11.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.513+0000 7f4841a2f640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2023-10-30T20:24:11.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.513+0000 7f4841a2f640 1 -- 172.21.15.69:0/1696255916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f483c141ce0 msgr2=0x7f483c0abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:11.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.513+0000 7f4841a2f640 1 --2- 172.21.15.69:0/1696255916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f483c141ce0 0x7f483c0abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.513+0000 7f4841a2f640 1 -- 172.21.15.69:0/1696255916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f483c0b4390 msgr2=0x7f483c1da300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:11.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.513+0000 7f4841a2f640 1 --2- 172.21.15.69:0/1696255916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f483c0b4390 0x7f483c1da300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.513+0000 7f4841a2f640 10 monclient: _finish_hunting 0 2023-10-30T20:24:11.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.513+0000 7f4841a2f640 1 monclient: found mon.noname-a 2023-10-30T20:24:11.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.513+0000 7f4841a2f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:11.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.513+0000 7f4841a2f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:11.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.513+0000 7f4841a2f640 1 -- 172.21.15.69:0/1696255916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c145800 con 0x7f483c1418a0 2023-10-30T20:24:11.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.517+0000 7f4841a2f640 1 --2- 172.21.15.69:0/1696255916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f483c1418a0 0x7f483c0aba10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f483800c4a0 tx=0x7f483800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:11.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.518+0000 7f4832ffd640 10 client.?.objecter ms_handle_connect 0x7f483c1418a0 2023-10-30T20:24:11.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.518+0000 7f4832ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:11.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4832ffd640 1 -- 172.21.15.69:0/1696255916 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4838005ce0 con 0x7f483c1418a0 2023-10-30T20:24:11.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4832ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:11.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4832ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:11.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4832ffd640 10 monclient: dump: 2023-10-30T20:24:11.535 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:11.535 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:11.535 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:11.535 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:11.535 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:11.536 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:11.536 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:11.536 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:11.536 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:11.536 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:11.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4832ffd640 10 monclient: _finish_auth 0 2023-10-30T20:24:11.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4832ffd640 10 monclient: _check_auth_tickets 2023-10-30T20:24:11.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4832ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:11.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4832ffd640 1 -- 172.21.15.69:0/1696255916 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483800e040 con 0x7f483c1418a0 2023-10-30T20:24:11.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4832ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:11.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4832ffd640 1 -- 172.21.15.69:0/1696255916 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4838005e90 con 0x7f483c1418a0 2023-10-30T20:24:11.538 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4832ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:11.538 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4832ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:11.538 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4832ffd640 10 monclient: dump: 2023-10-30T20:24:11.538 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:11.538 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:11.539 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:11.539 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:11.539 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:11.539 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:11.539 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:11.540 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:11.540 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:11.540 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:11.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4844494640 5 monclient: authenticate success, global_id 4353 2023-10-30T20:24:11.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4844494640 10 monclient: _renew_subs 2023-10-30T20:24:11.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4844494640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:11.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4840a2d640 4 set_mon_vals no callback set 2023-10-30T20:24:11.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4844494640 1 -- 172.21.15.69:0/1696255916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c0ab110 con 0x7f483c1418a0 2023-10-30T20:24:11.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4844494640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:11.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4844494640 10 monclient: _renew_subs 2023-10-30T20:24:11.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4844494640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:11.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4844494640 1 -- 172.21.15.69:0/1696255916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c0ac670 con 0x7f483c1418a0 2023-10-30T20:24:11.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.521+0000 7f4844494640 1 librados: init done 2023-10-30T20:24:11.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.522+0000 7f4844494640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:11.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.522+0000 7f4844494640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:11.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.522+0000 7f4844494640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:11.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.522+0000 7f4844494640 1 -- 172.21.15.69:0/1696255916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483c0a93d0 con 0x7f483c1418a0 2023-10-30T20:24:11.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.540+0000 7f4832ffd640 1 -- 172.21.15.69:0/1696255916 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f4838007270 con 0x7f483c1418a0 2023-10-30T20:24:11.544 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.540+0000 7f4832ffd640 1 --2- 172.21.15.69:0/1696255916 >> v2:172.21.15.69:6800/1876307353 conn(0x7f482006c840 0x7f482006ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:11.544 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.540+0000 7f4842230640 1 --2- 172.21.15.69:0/1696255916 >> v2:172.21.15.69:6800/1876307353 conn(0x7f482006c840 0x7f482006ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:11.544 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.540+0000 7f4842230640 10 monclient: get_auth_request con 0x7f482006c840 auth_method 0 2023-10-30T20:24:11.544 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.541+0000 7f4832ffd640 10 client.4353.objecter ms_handle_connect 0x7f482006c840 2023-10-30T20:24:11.544 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.541+0000 7f4842230640 1 --2- 172.21.15.69:0/1696255916 >> v2:172.21.15.69:6800/1876307353 conn(0x7f482006c840 0x7f482006ed00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f482c036000 tx=0x7f482c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:11.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.779+0000 7f4832ffd640 1 -- 172.21.15.69:0/1696255916 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f48380575e0 con 0x7f483c1418a0 2023-10-30T20:24:11.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.779+0000 7f4832ffd640 10 client.4353.objecter ms_dispatch 0x7f483c000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:11.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.779+0000 7f4832ffd640 3 client.4353.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:11.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.779+0000 7f4832ffd640 3 client.4353.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:11.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.779+0000 7f4832ffd640 20 client.4353.objecter dump_active .. 0 homeless 2023-10-30T20:24:11.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.779+0000 7f4832ffd640 1 -- 172.21.15.69:0/1696255916 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f483805b680 con 0x7f483c1418a0 2023-10-30T20:24:11.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.779+0000 7f4832ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:11.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.779+0000 7f4832ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:11.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.948+0000 7f4844494640 1 -- 172.21.15.69:0/1696255916 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f483c0aa3d0 con 0x7f482006c840 2023-10-30T20:24:11.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.950+0000 7f4832ffd640 1 -- 172.21.15.69:0/1696255916 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48061 (secure 0 0 0) 0x7f482c03a420 con 0x7f482006c840 2023-10-30T20:24:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:24:11.952 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-10-30T20:24:11.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 10 librados: watch_flush enter 2023-10-30T20:24:11.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 10 librados: watch_flush exit 2023-10-30T20:24:11.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 20 client.4353.objecter shutdown clearing up homeless session... 2023-10-30T20:24:11.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 10 client.4353.objecter successfully canceled tick 2023-10-30T20:24:11.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 1 -- 172.21.15.69:0/1696255916 >> v2:172.21.15.69:6800/1876307353 conn(0x7f482006c840 msgr2=0x7f482006ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:11.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 1 --2- 172.21.15.69:0/1696255916 >> v2:172.21.15.69:6800/1876307353 conn(0x7f482006c840 0x7f482006ed00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f482c036000 tx=0x7f482c03c040 comp rx=0 tx=0).stop 2023-10-30T20:24:11.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 10 monclient: shutdown 2023-10-30T20:24:11.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:11.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 1 -- 172.21.15.69:0/1696255916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f483c1418a0 msgr2=0x7f483c0aba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:11.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 1 --2- 172.21.15.69:0/1696255916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f483c1418a0 0x7f483c0aba10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f483800c4a0 tx=0x7f483800c960 comp rx=0 tx=0).stop 2023-10-30T20:24:11.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 1 -- 172.21.15.69:0/1696255916 shutdown_connections 2023-10-30T20:24:11.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 1 --2- 172.21.15.69:0/1696255916 >> v2:172.21.15.69:6800/1876307353 conn(0x7f482006c840 0x7f482006ed00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 1 --2- 172.21.15.69:0/1696255916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f483c141ce0 0x7f483c0abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 1 --2- 172.21.15.69:0/1696255916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f483c1418a0 0x7f483c0aba10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 1 --2- 172.21.15.69:0/1696255916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f483c0b4390 0x7f483c1da300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:11.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 1 -- 172.21.15.69:0/1696255916 >> 172.21.15.69:0/1696255916 conn(0x7f483c065ea0 msgr2=0x7f483c13a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:11.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 1 -- 172.21.15.69:0/1696255916 shutdown_connections 2023-10-30T20:24:11.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 1 -- 172.21.15.69:0/1696255916 wait complete. 2023-10-30T20:24:11.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:11.952+0000 7f4844494640 1 librados: shutdown 2023-10-30T20:24:11.964 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-30T20:24:10.172102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217700,"kb_used_data":1496,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757020,"statfs":{"total":773094113280,"available":772871188480,"internally_reserved":0,"allocated":1531904,"data_stored":1281464,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12745,"internal_metadata":221236791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.252186"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T20:24:06.843187+0000","last_change":"2023-10-30T20:24:04.700492+0000","last_active":"2023-10-30T20:24:06.843187+0000","last_peered":"2023-10-30T20:24:06.843187+0000","last_clean":"2023-10-30T20:24:06.843187+0000","last_became_active":"2023-10-30T20:24:04.692769+0000","last_became_peered":"2023-10-30T20:24:04.692769+0000","last_unstale":"2023-10-30T20:24:06.843187+0000","last_undegraded":"2023-10-30T20:24:06.843187+0000","last_fullsized":"2023-10-30T20:24:06.843187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:58.321494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T20:24:05.227061+0000","last_change":"2023-10-30T20:24:04.738314+0000","last_active":"2023-10-30T20:24:05.227061+0000","last_peered":"2023-10-30T20:24:05.227061+0000","last_clean":"2023-10-30T20:24:05.227061+0000","last_became_active":"2023-10-30T20:24:04.712006+0000","last_became_peered":"2023-10-30T20:24:04.712006+0000","last_unstale":"2023-10-30T20:24:05.227061+0000","last_undegraded":"2023-10-30T20:24:05.227061+0000","last_fullsized":"2023-10-30T20:24:05.227061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:41:08.136073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T20:24:06.848831+0000","last_change":"2023-10-30T20:24:04.315228+0000","last_active":"2023-10-30T20:24:06.848831+0000","last_peered":"2023-10-30T20:24:06.848831+0000","last_clean":"2023-10-30T20:24:06.848831+0000","last_became_active":"2023-10-30T20:24:04.310065+0000","last_became_peered":"2023-10-30T20:24:04.310065+0000","last_unstale":"2023-10-30T20:24:06.848831+0000","last_undegraded":"2023-10-30T20:24:06.848831+0000","last_fullsized":"2023-10-30T20:24:06.848831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:09:06.876832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T20:24:08.354070+0000","last_change":"2023-10-30T20:24:04.552803+0000","last_active":"2023-10-30T20:24:08.354070+0000","last_peered":"2023-10-30T20:24:08.354070+0000","last_clean":"2023-10-30T20:24:08.354070+0000","last_became_active":"2023-10-30T20:24:04.525673+0000","last_became_peered":"2023-10-30T20:24:04.525673+0000","last_unstale":"2023-10-30T20:24:08.354070+0000","last_undegraded":"2023-10-30T20:24:08.354070+0000","last_fullsized":"2023-10-30T20:24:08.354070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:18.627320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T20:24:06.822671+0000","last_change":"2023-10-30T20:24:04.525394+0000","last_active":"2023-10-30T20:24:06.822671+0000","last_peered":"2023-10-30T20:24:06.822671+0000","last_clean":"2023-10-30T20:24:06.822671+0000","last_became_active":"2023-10-30T20:24:04.512341+0000","last_became_peered":"2023-10-30T20:24:04.512341+0000","last_unstale":"2023-10-30T20:24:06.822671+0000","last_undegraded":"2023-10-30T20:24:06.822671+0000","last_fullsized":"2023-10-30T20:24:06.822671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:00:36.886515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T20:24:06.765719+0000","last_change":"2023-10-30T20:24:05.504482+0000","last_active":"2023-10-30T20:24:06.765719+0000","last_peered":"2023-10-30T20:24:06.765719+0000","last_clean":"2023-10-30T20:24:06.765719+0000","last_became_active":"2023-10-30T20:24:05.475394+0000","last_became_peered":"2023-10-30T20:24:05.475394+0000","last_unstale":"2023-10-30T20:24:06.765719+0000","last_undegraded":"2023-10-30T20:24:06.765719+0000","last_fullsized":"2023-10-30T20:24:06.765719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:21:02.335096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T20:24:06.810235+0000","last_change":"2023-10-30T20:24:04.284202+0000","last_active":"2023-10-30T20:24:06.810235+0000","last_peered":"2023-10-30T20:24:06.810235+0000","last_clean":"2023-10-30T20:24:06.810235+0000","last_became_active":"2023-10-30T20:24:04.254051+0000","last_became_peered":"2023-10-30T20:24:04.254051+0000","last_unstale":"2023-10-30T20:24:06.810235+0000","last_undegraded":"2023-10-30T20:24:06.810235+0000","last_fullsized":"2023-10-30T20:24:06.810235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:14:06.705421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T20:24:06.827697+0000","last_change":"2023-10-30T20:24:04.676966+0000","last_active":"2023-10-30T20:24:06.827697+0000","last_peered":"2023-10-30T20:24:06.827697+0000","last_clean":"2023-10-30T20:24:06.827697+0000","last_became_active":"2023-10-30T20:24:04.671790+0000","last_became_peered":"2023-10-30T20:24:04.671790+0000","last_unstale":"2023-10-30T20:24:06.827697+0000","last_undegraded":"2023-10-30T20:24:06.827697+0000","last_fullsized":"2023-10-30T20:24:06.827697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:56.600896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":117,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T20:24:06.245375+0000","last_change":"2023-10-30T20:23:57.127569+0000","last_active":"2023-10-30T20:24:06.245375+0000","last_peered":"2023-10-30T20:24:06.245375+0000","last_clean":"2023-10-30T20:24:06.245375+0000","last_became_active":"2023-10-30T20:23:57.098872+0000","last_became_peered":"2023-10-30T20:23:57.098872+0000","last_unstale":"2023-10-30T20:24:06.245375+0000","last_undegraded":"2023-10-30T20:24:06.245375+0000","last_fullsized":"2023-10-30T20:24:06.245375+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:23:54.534733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:23:54.534733+0000","last_clean_scrub_stamp":"2023-10-30T20:23:54.534733+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:26:29.039781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476742,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":59566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.202999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.4}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.534000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5860000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.058999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.420000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8529999999999998}]}]},{"osd":1,"up_from":16,"seq":68719476741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":457406,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9380000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.821000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0809999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.962}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.052}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.959}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.699}]}]},{"osd":0,"up_from":16,"seq":68719476743,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":59566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.934000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.865}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1370000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.205}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.896000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.763999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.702999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":64194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7360000000000007}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.309000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.368}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.782}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.501000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.173999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.126000000000001}]}]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":457406,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.113}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.249000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.219000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.042999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.590999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.012}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.459}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":188416,"data_stored":64194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.0720000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.515000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.59}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.314}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.420999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.542000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.56}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":59566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8019999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4339999999999993}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.241}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.031000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.128}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.343}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":34.646000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":59566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.148}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.48}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.056000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.321000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.673}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":177.608}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.788}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T20:24:11.965 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T20:24:11.966 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-30T20:24:11.966 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-30T20:24:11.966 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-30T20:24:12.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.063+0000 7f56bb116640 10 monclient: get_monmap_and_config 2023-10-30T20:24:12.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.063+0000 7f56bb116640 10 monclient: build_initial_monmap 2023-10-30T20:24:12.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.063+0000 7f56bb116640 10 monclient: monmap: 2023-10-30T20:24:12.065 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:12.065 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:12.065 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:24:12.063991+0000 2023-10-30T20:24:12.065 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:24:12.063991+0000 2023-10-30T20:24:12.065 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:12.066 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:12.066 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:24:12.066 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:24:12.066 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:24:12.066 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:24:12.067 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:24:12.067 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.063+0000 7f56bb116640 1 Processor -- start 2023-10-30T20:24:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.063+0000 7f56bb116640 1 -- start start 2023-10-30T20:24:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.063+0000 7f56bb116640 10 monclient: init 2023-10-30T20:24:12.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.064+0000 7f56bb116640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:12.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.064+0000 7f56bb116640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-10-30T20:24:12.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.064+0000 7f56bb116640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f56b40b0b40 0x7f56b4070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:12.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.064+0000 7f56bb116640 10 monclient(hunting): picked mon.noname-c con 0x7f56b40b0b40 addr v2:172.21.15.183:3300/0 2023-10-30T20:24:12.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.064+0000 7f56bb116640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f56b413ca20 0x7f56b413ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:12.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.064+0000 7f56bb116640 10 monclient(hunting): picked mon.noname-a con 0x7f56b413ca20 addr v2:172.21.15.69:3300/0 2023-10-30T20:24:12.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.064+0000 7f56bb116640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56b413f920 addr v1:172.21.15.183:6789/0 2023-10-30T20:24:12.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.064+0000 7f56bb116640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b4071590 con 0x7f56b413f920 2023-10-30T20:24:12.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.064+0000 7f56bb116640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:12.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.064+0000 7f56bb116640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b40679a0 con 0x7f56b413ca20 2023-10-30T20:24:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.064+0000 7f56bb116640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.064+0000 7f56bb116640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b40b0f20 con 0x7f56b40b0b40 2023-10-30T20:24:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.064+0000 7f56bb116640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.064+0000 7f56bb116640 10 monclient(hunting): authenticate will time out at 1575.353304s 2023-10-30T20:24:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.065+0000 7f56b8eb2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f56b40b0b40 0x7f56b4070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.065+0000 7f56b3fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f56b413ca20 0x7f56b413ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.065+0000 7f56b8eb2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f56b40b0b40 0x7f56b4070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:53210/0 (socket says 172.21.15.69:53210) 2023-10-30T20:24:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.065+0000 7f56b8eb2640 1 -- 172.21.15.69:0/3845884338 learned_addr learned my addr 172.21.15.69:0/3845884338 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.065+0000 7f56b8eb2640 10 monclient(hunting): get_auth_request con 0x7f56b40b0b40 auth_method 0 2023-10-30T20:24:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.065+0000 7f56b8eb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.065+0000 7f56b8eb2640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.065+0000 7f56b8eb2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.065+0000 7f56b3fff640 10 monclient(hunting): get_auth_request con 0x7f56b413ca20 auth_method 0 2023-10-30T20:24:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.065+0000 7f56b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.065+0000 7f56b3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:12.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.065+0000 7f56b3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:12.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.066+0000 7f56b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:12.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.066+0000 7f56b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:12.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.066+0000 7f56b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:12.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.066+0000 7f56b8eb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:12.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.066+0000 7f56b8eb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:12.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.066+0000 7f56b8eb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:12.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.067+0000 7f56b3fff640 10 monclient(hunting): handle_auth_done global_id 4356 payload 274 2023-10-30T20:24:12.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.067+0000 7f56b3fff640 1 -- 172.21.15.69:0/3845884338 >> v2:172.21.15.183:3300/0 conn(0x7f56b40b0b40 msgr2=0x7f56b4070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:24:12.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.067+0000 7f56b3fff640 1 --2- 172.21.15.69:0/3845884338 >> v2:172.21.15.183:3300/0 conn(0x7f56b40b0b40 0x7f56b4070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:12.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.067+0000 7f56b3fff640 1 -- 172.21.15.69:0/3845884338 >> v1:172.21.15.183:6789/0 conn(0x7f56b413f920 legacy=0x7f56b413fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:12.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.067+0000 7f56b3fff640 10 monclient: _finish_hunting 0 2023-10-30T20:24:12.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.067+0000 7f56b3fff640 1 monclient: found mon.noname-a 2023-10-30T20:24:12.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.067+0000 7f56b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:12.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.067+0000 7f56b3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:12.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.067+0000 7f56b3fff640 1 -- 172.21.15.69:0/3845884338 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b406d070 con 0x7f56b413ca20 2023-10-30T20:24:12.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.069+0000 7f56b3fff640 1 --2- 172.21.15.69:0/3845884338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b413ca20 0x7f56b413ce80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f56b4066a60 tx=0x7f56a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ada6ef60a090af server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:12.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56b37fe640 1 -- 172.21.15.69:0/3845884338 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a8031830 con 0x7f56b413ca20 2023-10-30T20:24:12.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:12.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56b37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:12.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56b37fe640 10 monclient: dump: 2023-10-30T20:24:12.082 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:12.083 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:12.083 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:12.083 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:12.083 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:12.083 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:12.084 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:12.084 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:12.084 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:12.084 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:12.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56b37fe640 10 monclient: _finish_auth 0 2023-10-30T20:24:12.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56b37fe640 10 monclient: _check_auth_tickets 2023-10-30T20:24:12.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:12.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56bb116640 5 monclient: authenticate success, global_id 4356 2023-10-30T20:24:12.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56b37fe640 1 -- 172.21.15.69:0/3845884338 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a802d040 con 0x7f56b413ca20 2023-10-30T20:24:12.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56bb116640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:24:12.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56b37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56bb116640 10 monclient: get_monmap_and_config success 2023-10-30T20:24:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56bb116640 4 set_mon_vals no callback set 2023-10-30T20:24:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56b37fe640 1 -- 172.21.15.69:0/3845884338 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a8031bc0 con 0x7f56b413ca20 2023-10-30T20:24:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:12.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56b37fe640 10 monclient: dump: 2023-10-30T20:24:12.087 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:12.087 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:12.087 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:12.087 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:12.089 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:12.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56bb116640 10 monclient: shutdown 2023-10-30T20:24:12.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56bb116640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:12.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56bb116640 1 -- 172.21.15.69:0/3845884338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b413ca20 msgr2=0x7f56b413ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:12.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56bb116640 1 --2- 172.21.15.69:0/3845884338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b413ca20 0x7f56b413ce80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f56b4066a60 tx=0x7f56a802b8d0 comp rx=0 tx=0).stop 2023-10-30T20:24:12.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56bb116640 1 -- 172.21.15.69:0/3845884338 shutdown_connections 2023-10-30T20:24:12.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56bb116640 1 --2- 172.21.15.69:0/3845884338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b413ca20 0x7f56b413ce80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:12.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56bb116640 1 --2- 172.21.15.69:0/3845884338 >> v2:172.21.15.183:3300/0 conn(0x7f56b40b0b40 0x7f56b4070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:12.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.073+0000 7f56bb116640 1 -- 172.21.15.69:0/3845884338 >> 172.21.15.69:0/3845884338 conn(0x7f56b40b4f20 msgr2=0x7f56b40b5330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:12.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.074+0000 7f56bb116640 1 -- 172.21.15.69:0/3845884338 shutdown_connections 2023-10-30T20:24:12.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.074+0000 7f56bb116640 1 -- 172.21.15.69:0/3845884338 wait complete. 2023-10-30T20:24:12.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.074+0000 7f56bb116640 10 monclient: build_initial_monmap 2023-10-30T20:24:12.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.074+0000 7f56bb116640 10 monclient: monmap: 2023-10-30T20:24:12.091 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:24:12.091 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:24:12.091 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:24:12.092 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:24:12.092 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:24:12.092 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:24:12.092 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:24:12.092 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:24:12.093 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:24:12.093 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:12.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.074+0000 7f56bb116640 1 librados: starting msgr at 2023-10-30T20:24:12.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.074+0000 7f56bb116640 1 librados: starting objecter 2023-10-30T20:24:12.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 1 Processor -- start 2023-10-30T20:24:12.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 1 -- start start 2023-10-30T20:24:12.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 1 librados: setting wanted keys 2023-10-30T20:24:12.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 1 librados: calling monclient init 2023-10-30T20:24:12.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 10 monclient: init 2023-10-30T20:24:12.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 10 monclient: _reopen_session rank -1 2023-10-30T20:24:12.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:24:12.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56b40b0b40 0x7f56b413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:12.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 10 monclient(hunting): picked mon.noname-b con 0x7f56b40b0b40 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:24:12.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56b413ca20 0x7f56b41da0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:12.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 10 monclient(hunting): picked mon.noname-c con 0x7f56b413ca20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:24:12.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b413f920 0x7f56b41da630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:12.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 10 monclient(hunting): picked mon.noname-a con 0x7f56b413f920 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:24:12.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:12.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40af180 con 0x7f56b413f920 2023-10-30T20:24:12.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:12.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b406afa0 con 0x7f56b40b0b40 2023-10-30T20:24:12.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 10 monclient(hunting): start opening mon connection 2023-10-30T20:24:12.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40aebb0 con 0x7f56b413ca20 2023-10-30T20:24:12.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 10 monclient(hunting): _renew_subs 2023-10-30T20:24:12.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.075+0000 7f56bb116640 10 monclient(hunting): authenticate will time out at 1575.364665s 2023-10-30T20:24:12.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b96b3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b413f920 0x7f56b41da630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:12.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56b413ca20 0x7f56b41da0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:12.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b8eb2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56b40b0b40 0x7f56b413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:12.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b96b3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b413f920 0x7f56b41da630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56816/0 (socket says 172.21.15.69:56816) 2023-10-30T20:24:12.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b96b3640 1 -- 172.21.15.69:0/3925839398 learned_addr learned my addr 172.21.15.69:0/3925839398 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:24:12.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b96b3640 10 monclient(hunting): get_auth_request con 0x7f56b413f920 auth_method 0 2023-10-30T20:24:12.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b96b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:12.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b96b3640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:12.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b96b3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:12.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b3fff640 10 monclient(hunting): get_auth_request con 0x7f56b413ca20 auth_method 0 2023-10-30T20:24:12.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:12.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:12.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:12.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b8eb2640 10 monclient(hunting): get_auth_request con 0x7f56b40b0b40 auth_method 0 2023-10-30T20:24:12.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b8eb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:24:12.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b8eb2640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:24:12.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.076+0000 7f56b8eb2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:24:12.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.077+0000 7f56b96b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:12.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.077+0000 7f56b96b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:12.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.077+0000 7f56b96b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:12.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.077+0000 7f56b8eb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:12.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.077+0000 7f56b8eb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:12.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.077+0000 7f56b8eb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:12.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.077+0000 7f56b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:24:12.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.077+0000 7f56b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:24:12.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.077+0000 7f56b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:24:12.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.079+0000 7f56b96b3640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2023-10-30T20:24:12.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.079+0000 7f56b96b3640 1 -- 172.21.15.69:0/3925839398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56b413ca20 msgr2=0x7f56b41da0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:12.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.079+0000 7f56b96b3640 1 --2- 172.21.15.69:0/3925839398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56b413ca20 0x7f56b41da0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:12.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.079+0000 7f56b96b3640 1 -- 172.21.15.69:0/3925839398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56b40b0b40 msgr2=0x7f56b413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:12.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.079+0000 7f56b96b3640 1 --2- 172.21.15.69:0/3925839398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56b40b0b40 0x7f56b413ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:12.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.079+0000 7f56b96b3640 10 monclient: _finish_hunting 0 2023-10-30T20:24:12.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.079+0000 7f56b96b3640 1 monclient: found mon.noname-a 2023-10-30T20:24:12.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.079+0000 7f56b96b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:24:12.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.079+0000 7f56b96b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:24:12.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.079+0000 7f56b96b3640 1 -- 172.21.15.69:0/3925839398 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b40b3570 con 0x7f56b413f920 2023-10-30T20:24:12.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.082+0000 7f56b96b3640 1 --2- 172.21.15.69:0/3925839398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b413f920 0x7f56b41da630 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00c730 tx=0x7f56ac00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:12.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.082+0000 7f56b17fa640 10 client.?.objecter ms_handle_connect 0x7f56b413f920 2023-10-30T20:24:12.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.082+0000 7f56b17fa640 10 client.?.objecter resend_mon_ops 2023-10-30T20:24:12.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56b17fa640 1 -- 172.21.15.69:0/3925839398 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac003b70 con 0x7f56b413f920 2023-10-30T20:24:12.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:12.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56b17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:24:12.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56b17fa640 10 monclient: dump: 2023-10-30T20:24:12.107 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:12.107 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:12.107 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:12.107 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:12.108 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:12.108 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:12.108 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:12.108 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:12.108 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:12.109 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:12.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56b17fa640 10 monclient: _finish_auth 0 2023-10-30T20:24:12.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56b17fa640 10 monclient: _check_auth_tickets 2023-10-30T20:24:12.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:24:12.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56b17fa640 1 -- 172.21.15.69:0/3925839398 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ac00e040 con 0x7f56b413f920 2023-10-30T20:24:12.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56b17fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:24:12.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56b17fa640 1 -- 172.21.15.69:0/3925839398 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac0050d0 con 0x7f56b413f920 2023-10-30T20:24:12.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:24:12.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:24:12.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56b17fa640 10 monclient: dump: 2023-10-30T20:24:12.111 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:24:12.111 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:24:12.111 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:12.111 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:24:12.111 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:24:12.111 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:24:12.112 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:24:12.112 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:24:12.112 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:24:12.112 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:24:12.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56bb116640 5 monclient: authenticate success, global_id 4359 2023-10-30T20:24:12.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56bb116640 10 monclient: _renew_subs 2023-10-30T20:24:12.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56bb116640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:12.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56b2ffd640 4 set_mon_vals no callback set 2023-10-30T20:24:12.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56bb116640 1 -- 172.21.15.69:0/3925839398 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b41dab70 con 0x7f56b413f920 2023-10-30T20:24:12.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56bb116640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:24:12.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56bb116640 10 monclient: _renew_subs 2023-10-30T20:24:12.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56bb116640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:12.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56bb116640 1 -- 172.21.15.69:0/3925839398 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b406b7a0 con 0x7f56b413f920 2023-10-30T20:24:12.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.085+0000 7f56bb116640 1 librados: init done 2023-10-30T20:24:12.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.086+0000 7f56bb116640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:12.115 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.086+0000 7f56bb116640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:12.115 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.086+0000 7f56bb116640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:12.115 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.086+0000 7f56bb116640 1 -- 172.21.15.69:0/3925839398 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b404f050 con 0x7f56b413f920 2023-10-30T20:24:12.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.324+0000 7f56b17fa640 1 -- 172.21.15.69:0/3925839398 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f56ac005240 con 0x7f56b413f920 2023-10-30T20:24:12.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.324+0000 7f56b17fa640 1 --2- 172.21.15.69:0/3925839398 >> v2:172.21.15.69:6800/1876307353 conn(0x7f56a006c810 0x7f56a006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:24:12.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.324+0000 7f56b8eb2640 1 --2- 172.21.15.69:0/3925839398 >> v2:172.21.15.69:6800/1876307353 conn(0x7f56a006c810 0x7f56a006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:24:12.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.324+0000 7f56b8eb2640 10 monclient: get_auth_request con 0x7f56a006c810 auth_method 0 2023-10-30T20:24:12.327 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.324+0000 7f56b8eb2640 1 --2- 172.21.15.69:0/3925839398 >> v2:172.21.15.69:6800/1876307353 conn(0x7f56a006c810 0x7f56a006ecd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f56a4009bd0 tx=0x7f56a4009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:24:12.327 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.324+0000 7f56b17fa640 10 client.4359.objecter ms_handle_connect 0x7f56a006c810 2023-10-30T20:24:12.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.350+0000 7f56b17fa640 1 -- 172.21.15.69:0/3925839398 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f56ac058550 con 0x7f56b413f920 2023-10-30T20:24:12.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.350+0000 7f56b17fa640 10 client.4359.objecter ms_dispatch 0x7f56b4000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T20:24:12.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.350+0000 7f56b17fa640 3 client.4359.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:24:12.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.350+0000 7f56b17fa640 3 client.4359.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:24:12.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.350+0000 7f56b17fa640 20 client.4359.objecter dump_active .. 0 homeless 2023-10-30T20:24:12.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.350+0000 7f56b17fa640 1 -- 172.21.15.69:0/3925839398 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f56ac05c5f0 con 0x7f56b413f920 2023-10-30T20:24:12.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.350+0000 7f56b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:24:12.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.350+0000 7f56b17fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:24:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.605+0000 7f56bb116640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-10-30T20:24:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.605+0000 7f56bb116640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-10-30T20:24:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.605+0000 7f56bb116640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:24:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.605+0000 7f56bb116640 1 -- 172.21.15.69:0/3925839398 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f56b41dab70 con 0x7f56b413f920 2023-10-30T20:24:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.612+0000 7f56b17fa640 1 -- 172.21.15.69:0/3925839398 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f56ac0607a0 con 0x7f56b413f920 2023-10-30T20:24:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.612+0000 7f56b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-10-30T20:24:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.612+0000 7f56b17fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:24:12.614 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:24:12.615 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-30T20:24:12.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 10 librados: watch_flush enter 2023-10-30T20:24:12.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 10 librados: watch_flush exit 2023-10-30T20:24:12.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 20 client.4359.objecter shutdown clearing up homeless session... 2023-10-30T20:24:12.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 10 client.4359.objecter successfully canceled tick 2023-10-30T20:24:12.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 1 -- 172.21.15.69:0/3925839398 >> v2:172.21.15.69:6800/1876307353 conn(0x7f56a006c810 msgr2=0x7f56a006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:12.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 1 --2- 172.21.15.69:0/3925839398 >> v2:172.21.15.69:6800/1876307353 conn(0x7f56a006c810 0x7f56a006ecd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f56a4009bd0 tx=0x7f56a4009240 comp rx=0 tx=0).stop 2023-10-30T20:24:12.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 10 monclient: shutdown 2023-10-30T20:24:12.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:24:12.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 1 -- 172.21.15.69:0/3925839398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b413f920 msgr2=0x7f56b41da630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:24:12.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 1 --2- 172.21.15.69:0/3925839398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b413f920 0x7f56b41da630 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00c730 tx=0x7f56ac00cbf0 comp rx=0 tx=0).stop 2023-10-30T20:24:12.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 1 -- 172.21.15.69:0/3925839398 shutdown_connections 2023-10-30T20:24:12.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 1 --2- 172.21.15.69:0/3925839398 >> v2:172.21.15.69:6800/1876307353 conn(0x7f56a006c810 0x7f56a006ecd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:12.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 1 --2- 172.21.15.69:0/3925839398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56b413f920 0x7f56b41da630 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:12.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 1 --2- 172.21.15.69:0/3925839398 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56b413ca20 0x7f56b41da0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:12.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 1 --2- 172.21.15.69:0/3925839398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56b40b0b40 0x7f56b413ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:24:12.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 1 -- 172.21.15.69:0/3925839398 >> 172.21.15.69:0/3925839398 conn(0x7f56b40b4f20 msgr2=0x7f56b4136770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:24:12.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 1 -- 172.21.15.69:0/3925839398 shutdown_connections 2023-10-30T20:24:12.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 1 -- 172.21.15.69:0/3925839398 wait complete. 2023-10-30T20:24:12.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:24:12.615+0000 7f56bb116640 1 librados: shutdown 2023-10-30T20:24:12.628 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-30T20:24:12.628 INFO:teuthology.run_tasks:Running task workunit... 2023-10-30T20:24:12.641 INFO:tasks.workunit:Pulling workunits from ref d0db9f2f95b9d6344950185071cace2a50d296eb 2023-10-30T20:24:12.642 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-30T20:24:12.642 DEBUG:teuthology.orchestra.run.smithi069:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-30T20:24:12.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T20:24:12.657 INFO:teuthology.orchestra.run.smithi069.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-30T20:24:12.657 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-30T20:24:12.711 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-30T20:24:12.712 DEBUG:teuthology.orchestra.run.smithi069:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-30T20:24:12.765 INFO:tasks.workunit:timeout=6h 2023-10-30T20:24:12.765 INFO:tasks.workunit:cleanup=True 2023-10-30T20:24:12.766 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d0db9f2f95b9d6344950185071cace2a50d296eb 2023-10-30T20:24:12.820 INFO:tasks.workunit.client.0.smithi069.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-30T20:24:23.097 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:45.156 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:23.144 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:45.203 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:23.182 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:45.241 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:23.190 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:45.249 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:23.196 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:45.255 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:23.197 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:45.224 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:23.199 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:45.226 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:23.236 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:45.264 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:23.242 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:45.301 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:43.141 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:05.199 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:43.192 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:05.251 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:43.215 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:05.273 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:43.241 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:05.268 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:43.243 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:05.270 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:43.254 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:05.313 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:43.258 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:05.286 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:43.260 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:05.319 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:24:43.330 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:05.389 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:03.208 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:25.267 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:03.257 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:25.316 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:03.274 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:25.332 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:03.309 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:25.336 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:03.311 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:25.338 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:03.324 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:25.351 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:03.335 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:25.394 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:03.340 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:25.399 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:03.364 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:25.423 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:15.632 INFO:tasks.workunit.client.0.smithi069.stderr:Note: switching to 'd0db9f2f95b9d6344950185071cace2a50d296eb'. 2023-10-30T20:25:15.632 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-10-30T20:25:15.633 INFO:tasks.workunit.client.0.smithi069.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-30T20:25:15.633 INFO:tasks.workunit.client.0.smithi069.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-30T20:25:15.633 INFO:tasks.workunit.client.0.smithi069.stderr:state without impacting any branches by switching back to a branch. 2023-10-30T20:25:15.633 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-10-30T20:25:15.633 INFO:tasks.workunit.client.0.smithi069.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-30T20:25:15.634 INFO:tasks.workunit.client.0.smithi069.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-30T20:25:15.634 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-10-30T20:25:15.634 INFO:tasks.workunit.client.0.smithi069.stderr: git switch -c 2023-10-30T20:25:15.634 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-10-30T20:25:15.635 INFO:tasks.workunit.client.0.smithi069.stderr:Or undo this operation with: 2023-10-30T20:25:15.635 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-10-30T20:25:15.635 INFO:tasks.workunit.client.0.smithi069.stderr: git switch - 2023-10-30T20:25:15.635 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-10-30T20:25:15.635 INFO:tasks.workunit.client.0.smithi069.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-30T20:25:15.636 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-10-30T20:25:15.636 INFO:tasks.workunit.client.0.smithi069.stderr:HEAD is now at d0db9f2f95b Merge pull request #54082 from pdvian/wip-qa-suites-fix 2023-10-30T20:25:15.637 DEBUG:teuthology.orchestra.run.smithi069:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-30T20:25:15.699 INFO:tasks.workunit.client.0.smithi069.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-30T20:25:15.701 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T20:25:15.701 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-30T20:25:16.355 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-30T20:25:16.430 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-30T20:25:16.470 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T20:25:16.471 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T20:25:16.472 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-30T20:25:16.512 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T20:25:16.515 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-30T20:25:16.515 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-30T20:25:16.570 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-10-30T20:25:16.571 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-10-30T20:25:16.571 DEBUG:teuthology.orchestra.run.smithi069:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d0db9f2f95b9d6344950185071cace2a50d296eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-10-30T20:25:16.629 INFO:tasks.workunit.client.0.smithi069.stderr:+ parallel=1 2023-10-30T20:25:16.630 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' '' = --serial ']' 2023-10-30T20:25:16.630 INFO:tasks.workunit.client.0.smithi069.stderr:+ color= 2023-10-30T20:25:16.630 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -t 1 ']' 2023-10-30T20:25:16.630 INFO:tasks.workunit.client.0.smithi069.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-10-30T20:25:16.630 INFO:tasks.workunit.client.0.smithi069.stderr:+ declare -A pids 2023-10-30T20:25:16.631 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:16.631 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:16.631 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_aio 2023-10-30T20:25:16.631 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_aio' 2023-10-30T20:25:16.631 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_aio 2023-10-30T20:25:16.632 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:16.632 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_aio 2023-10-30T20:25:16.632 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91478 2023-10-30T20:25:16.633 INFO:tasks.workunit.client.0.smithi069.stdout:test api_aio on pid 91478 2023-10-30T20:25:16.635 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_aio on pid 91478' 2023-10-30T20:25:16.636 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91478 2023-10-30T20:25:16.636 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:16.636 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:16.637 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-10-30T20:25:16.637 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_aio_pp 2023-10-30T20:25:16.637 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_aio_pp' 2023-10-30T20:25:16.637 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_aio_pp 2023-10-30T20:25:16.637 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:16.638 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:16.638 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:16.638 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:16.638 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:16.638 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.639 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:16.639 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:16.639 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:16.639 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:16.639 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.639 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:16.640 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.640 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:16.640 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:16.640 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.640 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:16.641 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.641 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:16.641 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.641 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.641 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.642 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:16.642 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.642 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:16.642 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.642 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_aio_pp 2023-10-30T20:25:16.642 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91485 2023-10-30T20:25:16.643 INFO:tasks.workunit.client.0.smithi069.stdout:test api_aio_pp on pid 91485 2023-10-30T20:25:16.646 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_aio_pp on pid 91485' 2023-10-30T20:25:16.647 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91485 2023-10-30T20:25:16.647 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:16.649 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-10-30T20:25:16.649 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:16.649 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_io 2023-10-30T20:25:16.649 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_io' 2023-10-30T20:25:16.650 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_io 2023-10-30T20:25:16.650 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:16.650 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:16.650 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:16.650 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:16.651 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:16.651 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.651 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:16.651 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:16.651 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:16.651 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:16.652 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.652 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:16.652 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.652 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:16.652 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:16.653 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.653 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:16.653 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.653 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:16.653 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.653 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.654 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.654 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:16.654 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.654 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:16.654 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.655 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:16.655 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:16.655 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:16.655 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.655 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:16.655 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.656 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:16.656 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:16.656 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:16.656 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.656 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:16.657 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.657 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:16.657 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:16.657 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:16.657 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.657 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:16.658 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.658 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:16.658 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.658 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_io 2023-10-30T20:25:16.658 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91494 2023-10-30T20:25:16.659 INFO:tasks.workunit.client.0.smithi069.stdout:test api_io on pid 91494 2023-10-30T20:25:16.663 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_io on pid 91494' 2023-10-30T20:25:16.663 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91494 2023-10-30T20:25:16.664 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:16.664 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:16.664 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2023-10-30T20:25:16.664 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_io_pp 2023-10-30T20:25:16.664 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_io_pp' 2023-10-30T20:25:16.665 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_io_pp 2023-10-30T20:25:16.665 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:16.665 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:16.665 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:16.665 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:16.666 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:16.666 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:16.666 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:16.666 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:16.666 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.666 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:16.667 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.667 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.667 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:16.667 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:16.667 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:16.668 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:16.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:16.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:16.669 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:16.669 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:16.669 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.669 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:16.669 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.669 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:16.670 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.670 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:16.670 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:16.670 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:16.670 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.671 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:16.671 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:16.671 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.671 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:16.671 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:16.671 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:16.672 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.672 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:16.672 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.672 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:16.672 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:16.673 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.673 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:16.673 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:16.673 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.673 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.674 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:16.674 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.674 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.674 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.674 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:16.674 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.675 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:16.675 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.675 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_io_pp 2023-10-30T20:25:16.675 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91505 2023-10-30T20:25:16.676 INFO:tasks.workunit.client.0.smithi069.stdout:test api_io_pp on pid 91505 2023-10-30T20:25:16.681 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_io_pp on pid 91505' 2023-10-30T20:25:16.681 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91505 2023-10-30T20:25:16.681 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:16.682 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:16.682 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2023-10-30T20:25:16.682 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_asio 2023-10-30T20:25:16.682 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_asio' 2023-10-30T20:25:16.682 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_asio 2023-10-30T20:25:16.683 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:16.683 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:16.683 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:16.683 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:16.683 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:16.684 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.684 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:16.684 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:16.684 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:16.684 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:16.684 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:16.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:16.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:16.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:16.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:16.686 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:16.686 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.686 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.686 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:16.686 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.687 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:16.687 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:16.687 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.687 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:16.687 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.688 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:16.688 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.688 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.688 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.688 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:16.689 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.689 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:16.689 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.689 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:16.689 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:16.689 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:16.690 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.690 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:16.690 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.690 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:16.690 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:16.691 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:16.691 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.691 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:16.691 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.691 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:16.691 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:16.692 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:16.692 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.692 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:16.692 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.692 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:16.693 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.693 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:16.693 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:16.693 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:16.693 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.693 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:16.694 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.694 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:16.694 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:16.694 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.694 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:16.695 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.695 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:16.695 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.695 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:16.695 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.695 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:16.696 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:16.696 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_asio 2023-10-30T20:25:16.696 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:16.696 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91519 2023-10-30T20:25:16.697 INFO:tasks.workunit.client.0.smithi069.stdout:test api_asio on pid 91519 2023-10-30T20:25:16.703 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_asio on pid 91519' 2023-10-30T20:25:16.703 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91519 2023-10-30T20:25:16.703 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:16.703 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:16.704 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2023-10-30T20:25:16.704 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_list 2023-10-30T20:25:16.704 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_list' 2023-10-30T20:25:16.704 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.704 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_list 2023-10-30T20:25:16.704 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:16.705 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:16.705 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.705 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:16.705 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:16.705 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:16.706 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:16.706 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:16.706 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:16.706 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:16.706 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.707 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:16.707 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.707 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:16.707 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:16.707 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:16.707 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.708 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:16.708 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:16.708 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:16.708 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.708 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:16.709 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.709 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:16.709 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:16.709 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:16.709 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:16.709 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.710 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:16.710 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.710 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:16.710 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:16.710 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.711 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:16.711 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.711 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.711 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:16.711 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:16.712 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:16.712 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:16.712 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:16.712 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:16.712 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.712 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:16.713 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.713 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:16.713 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:16.713 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:16.713 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:16.714 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.714 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:16.714 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:16.714 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.714 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:16.715 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:16.715 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:16.715 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.715 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.715 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:16.715 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:16.716 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:16.716 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.716 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.716 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:16.716 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:16.717 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.717 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:16.717 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.717 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:16.717 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.717 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:16.718 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.718 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.718 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.718 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:16.718 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.719 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:16.719 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.719 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:16.719 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:16.719 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:16.719 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.720 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:16.720 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.720 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:16.720 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:16.720 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.721 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.721 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:16.721 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.721 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:16.721 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.721 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:16.722 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:16.722 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.722 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:16.722 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.722 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:16.723 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_list 2023-10-30T20:25:16.723 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:16.723 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91535 2023-10-30T20:25:16.723 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.723 INFO:tasks.workunit.client.0.smithi069.stdout:test api_list on pid 91535 2023-10-30T20:25:16.729 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_list on pid 91535' 2023-10-30T20:25:16.730 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:16.730 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91535 2023-10-30T20:25:16.730 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2023-10-30T20:25:16.730 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:16.730 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:16.731 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.731 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:16.731 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:16.731 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_lock 2023-10-30T20:25:16.731 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91478/exe 2023-10-30T20:25:16.731 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:16.732 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_lock' 2023-10-30T20:25:16.732 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_lock 2023-10-30T20:25:16.732 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:16.732 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:16.732 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:16.733 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:16.733 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:16.733 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:16.733 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:16.733 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:16.733 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:16.734 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.734 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.734 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:16.734 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:16.734 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.735 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:16.735 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:16.735 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:16.735 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.735 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:16.735 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:16.736 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:16.736 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.736 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:16.736 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:16.736 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:16.737 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:16.737 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.737 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:16.737 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:16.737 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.738 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.738 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:16.738 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:16.738 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:16.738 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.738 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.739 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:16.739 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:16.739 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.739 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:16.739 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.740 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:16.740 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.740 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:16.740 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:16.740 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.740 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:16.741 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:16.741 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:16.741 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:16.741 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.741 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:16.742 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:16.742 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.742 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:16.742 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.742 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:16.742 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.743 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:16.743 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.743 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:16.743 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:16.743 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.744 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.744 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:16.744 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:16.744 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.744 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:16.744 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:16.745 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:16.745 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.745 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:16.745 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.745 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:16.746 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:16.746 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:16.746 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:16.746 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.746 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:16.746 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:16.747 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:16.747 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.747 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:16.747 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.747 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:16.748 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.748 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:16.748 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:16.748 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.748 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:16.748 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.749 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.749 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:16.749 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.749 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:16.749 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:16.750 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.750 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:16.750 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:16.750 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:16.750 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.750 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:16.751 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:16.751 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:16.751 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91485/exe 2023-10-30T20:25:16.751 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:16.751 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_aio 2023-10-30T20:25:16.752 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_lock 2023-10-30T20:25:16.752 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_aio.log 2023-10-30T20:25:16.752 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91557 2023-10-30T20:25:16.752 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_lock on pid 91557' 2023-10-30T20:25:16.752 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.752 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:16.753 INFO:tasks.workunit.client.0.smithi069.stdout:test api_lock on pid 91557 2023-10-30T20:25:16.760 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.761 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91557 2023-10-30T20:25:16.761 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:16.761 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:16.761 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:16.761 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:16.762 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:16.762 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_aio: /' 2023-10-30T20:25:16.762 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2023-10-30T20:25:16.762 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_lock_pp 2023-10-30T20:25:16.762 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_lock_pp' 2023-10-30T20:25:16.763 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:16.763 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:16.763 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:16.763 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:16.763 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:16.763 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:16.764 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.764 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:16.764 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:16.764 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.764 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:16.765 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:16.765 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:16.765 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:16.765 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:16.765 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.765 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:16.766 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.766 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:16.766 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:16.766 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:16.766 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:16.767 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:16.767 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:16.767 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:16.767 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:16.767 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.767 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:16.768 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.768 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:16.768 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:16.768 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:16.768 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.769 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:16.769 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:16.769 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:16.769 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:16.769 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.769 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:16.770 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.770 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:16.770 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:16.770 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.770 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:16.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:16.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:16.771 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:16.771 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:16.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:16.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:16.772 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:16.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:16.773 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.773 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:16.773 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:16.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:16.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.774 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:16.774 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.774 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:16.774 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:16.774 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_aio_pp.log 2023-10-30T20:25:16.775 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_lock_pp 2023-10-30T20:25:16.775 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_aio_pp 2023-10-30T20:25:16.775 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:16.775 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:16.775 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:16.775 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:16.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:16.776 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:16.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.777 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:16.777 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:16.777 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:16.777 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:16.777 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.777 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.778 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:16.778 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.778 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:16.778 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.778 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.779 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.779 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:16.779 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.779 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:16.779 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.780 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_aio_pp: /' 2023-10-30T20:25:16.780 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:16.780 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.780 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:16.780 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:16.780 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.781 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.781 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.781 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:16.781 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:16.781 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:16.781 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.782 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:16.782 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.782 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:16.783 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:16.783 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:16.783 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.783 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:16.783 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.784 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:16.784 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:16.784 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91494/exe 2023-10-30T20:25:16.784 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:16.785 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_lock_pp 2023-10-30T20:25:16.785 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:16.785 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:16.785 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:16.785 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:16.786 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.786 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:16.786 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:16.786 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91579 2023-10-30T20:25:16.787 INFO:tasks.workunit.client.0.smithi069.stdout:test api_lock_pp on pid 91579 2023-10-30T20:25:16.792 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_lock_pp on pid 91579' 2023-10-30T20:25:16.793 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.793 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:16.793 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91579 2023-10-30T20:25:16.793 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.793 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:16.794 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:16.794 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:16.794 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.794 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:16.794 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:16.794 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.795 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:16.795 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.795 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.795 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:16.795 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2023-10-30T20:25:16.796 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:16.796 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_misc 2023-10-30T20:25:16.796 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.796 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:16.796 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.797 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:16.797 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:16.797 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_misc' 2023-10-30T20:25:16.797 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.797 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:16.797 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.798 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:16.798 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:16.798 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_misc 2023-10-30T20:25:16.798 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:16.799 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:16.799 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:16.799 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:16.799 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:16.799 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.800 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:16.800 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.800 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:16.800 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:16.800 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:16.801 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:16.801 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:16.801 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:16.801 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:16.801 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.801 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:16.802 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.802 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:16.802 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:16.802 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:16.802 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:16.803 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:16.803 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:16.803 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:16.803 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.803 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:16.803 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:16.804 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.804 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:16.804 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:16.804 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:16.804 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.805 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.805 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:16.805 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:16.805 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:16.805 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:16.806 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:16.806 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_io 2023-10-30T20:25:16.806 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:16.806 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.806 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:16.807 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.807 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:16.807 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:16.807 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.807 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:16.808 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:16.808 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.808 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:16.808 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.808 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_misc 2023-10-30T20:25:16.808 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:16.809 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:16.809 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.809 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.809 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:16.809 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.810 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:16.810 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91593 2023-10-30T20:25:16.810 INFO:tasks.workunit.client.0.smithi069.stdout:test api_misc on pid 91593 2023-10-30T20:25:16.816 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_misc on pid 91593' 2023-10-30T20:25:16.816 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:16.816 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91593 2023-10-30T20:25:16.816 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.816 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:16.817 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:16.817 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.817 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:16.817 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:16.817 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_io: /' 2023-10-30T20:25:16.818 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.818 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:16.818 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.818 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.819 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:16.819 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.819 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:16.819 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:16.819 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:16.819 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_misc_pp 2023-10-30T20:25:16.820 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.820 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_misc_pp' 2023-10-30T20:25:16.820 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2023-10-30T20:25:16.820 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_misc_pp 2023-10-30T20:25:16.821 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:16.821 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_io.log 2023-10-30T20:25:16.821 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91505/exe 2023-10-30T20:25:16.821 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:16.821 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:16.822 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:16.822 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:16.822 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:16.822 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:16.823 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.823 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:16.823 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:16.823 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:16.823 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:16.823 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.824 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:16.824 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.824 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:16.824 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:16.824 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.825 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:16.825 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.825 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:16.825 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.826 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:16.826 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.826 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:16.826 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:16.827 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:16.827 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:16.827 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:16.827 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:16.828 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:16.828 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:16.828 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:16.828 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:16.828 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:16.829 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:16.829 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:16.829 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:16.829 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.829 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:16.830 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.830 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:16.830 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:16.831 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.831 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:16.831 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:16.831 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:16.831 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_misc_pp 2023-10-30T20:25:16.832 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91610 2023-10-30T20:25:16.832 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_misc_pp on pid 91610' 2023-10-30T20:25:16.832 INFO:tasks.workunit.client.0.smithi069.stdout:test api_misc_pp on pid 91610 2023-10-30T20:25:16.856 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2023-10-30T20:25:16.857 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91610 2023-10-30T20:25:16.857 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:16.857 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:16.857 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_io_pp.log 2023-10-30T20:25:16.858 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:16.858 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:16.858 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.858 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:16.858 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.859 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:16.859 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:16.859 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.859 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:16.859 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_io_pp: /' 2023-10-30T20:25:16.860 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.860 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:16.860 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.861 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:16.861 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:16.861 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.861 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:16.861 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:16.862 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.862 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:16.862 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.862 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:16.862 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.863 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.863 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:16.863 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:16.863 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.863 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:16.864 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.864 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:16.864 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:16.864 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.864 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_io_pp 2023-10-30T20:25:16.864 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:16.865 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:16.865 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:16.865 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:16.865 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:16.865 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.866 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:16.866 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.866 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:16.866 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:16.866 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:16.867 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.867 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:16.867 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.867 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:16.867 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:16.867 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:16.868 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:16.868 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.868 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:16.868 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.868 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:16.869 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.869 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.869 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:16.869 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:16.869 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:16.870 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.870 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:16.870 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.870 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:16.870 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:16.871 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:16.871 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.871 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:16.871 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.871 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:16.871 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.872 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.872 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.872 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:16.872 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.872 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:16.873 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.873 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:16.873 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:16.873 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:16.873 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.873 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:16.874 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.874 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:16.874 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:16.874 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.874 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.875 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:16.875 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.875 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:16.875 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.875 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:16.876 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:16.876 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.876 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:16.876 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:16.876 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.877 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:16.877 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:16.877 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:16.877 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:16.877 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.877 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:16.878 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:16.878 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.878 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:16.878 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:16.878 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.879 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:16.879 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:16.879 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.879 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.879 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:16.880 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.880 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:16.880 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.880 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:16.880 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:16.880 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:16.881 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.881 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:16.881 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.881 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:16.881 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:16.882 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.882 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:16.882 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:16.882 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:16.882 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.883 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:16.883 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.883 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:16.883 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:16.883 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:16.884 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.884 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:16.884 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.884 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:16.884 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:16.885 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:16.885 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.885 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:16.885 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.885 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91535/exe 2023-10-30T20:25:16.885 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:16.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:16.886 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:16.886 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.886 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:16.887 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:16.887 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:16.887 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.887 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:16.887 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.887 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:16.888 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:16.888 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:16.888 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:16.888 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.888 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:16.888 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.889 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:16.889 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:16.889 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:16.889 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.889 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:16.890 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.890 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:16.890 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.890 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:16.890 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:16.891 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:16.891 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:16.891 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:16.891 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:16.891 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:16.891 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:16.892 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:16.892 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:16.892 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.892 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:16.892 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.893 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:16.893 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:16.893 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:16.893 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.893 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:16.894 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.894 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:16.894 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:16.894 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:16.894 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.895 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:16.895 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.895 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:16.895 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:16.895 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:16.896 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:16.896 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:16.896 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.896 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_list.log 2023-10-30T20:25:16.896 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:16.897 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_list: /' 2023-10-30T20:25:16.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:16.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:16.897 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_list 2023-10-30T20:25:16.897 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:16.898 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.898 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:16.898 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.898 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:16.898 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.899 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:16.899 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.899 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:16.899 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:16.899 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:16.900 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_tier_pp 2023-10-30T20:25:16.900 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_tier_pp' 2023-10-30T20:25:16.900 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.900 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:16.900 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.901 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:16.901 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:16.901 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:16.901 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:16.901 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:16.901 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:16.902 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:16.902 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.902 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:16.902 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.902 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:16.903 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:16.903 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.903 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:16.903 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:16.903 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:16.904 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.904 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:16.904 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.904 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:16.904 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:16.905 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:16.905 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:16.905 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.905 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:16.905 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.905 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:16.906 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:16.906 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.906 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91519/exe 2023-10-30T20:25:16.906 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:16.906 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.907 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.907 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:16.907 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.907 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:16.907 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.908 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:16.908 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:16.908 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.908 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:16.908 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.909 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.909 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:16.909 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:16.909 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.909 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:16.909 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:16.910 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:16.910 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:16.910 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_tier_pp 2023-10-30T20:25:16.910 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:16.910 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_tier_pp 2023-10-30T20:25:16.911 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91648 2023-10-30T20:25:16.911 INFO:tasks.workunit.client.0.smithi069.stdout:test api_tier_pp on pid 91648 2023-10-30T20:25:16.927 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_tier_pp on pid 91648' 2023-10-30T20:25:16.927 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91648 2023-10-30T20:25:16.927 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:16.927 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:16.928 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:16.928 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:16.928 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:16.928 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:16.928 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:16.929 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:16.929 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:16.929 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:16.929 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:16.929 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:16.929 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:16.930 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:16.930 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:16.930 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.930 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:16.930 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.931 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:16.931 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:16.931 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_asio 2023-10-30T20:25:16.931 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.931 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.932 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:16.932 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_asio.log 2023-10-30T20:25:16.932 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.932 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:16.932 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.933 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:16.933 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:16.933 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.933 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:16.933 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.933 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:16.934 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:16.934 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91579/exe 2023-10-30T20:25:16.934 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:16.934 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_pool 2023-10-30T20:25:16.934 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_pool' 2023-10-30T20:25:16.935 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:16.935 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:16.935 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:16.935 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.935 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:16.936 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.936 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:16.936 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:16.936 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.936 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:16.937 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.937 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:16.937 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:16.937 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.937 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:16.937 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.938 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:16.938 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:16.938 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:16.938 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:16.938 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:16.939 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:16.939 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:16.939 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:16.939 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:16.939 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:16.939 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:16.940 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_asio: /' 2023-10-30T20:25:16.940 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_lock_pp 2023-10-30T20:25:16.940 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_lock_pp.log 2023-10-30T20:25:16.940 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2023-10-30T20:25:16.940 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:16.941 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:16.941 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:16.941 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.941 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:16.941 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.941 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:16.942 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:16.942 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.942 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:16.942 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.942 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:16.943 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.943 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:16.943 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.943 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:16.943 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:16.944 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:16.944 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:16.944 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:16.944 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:16.944 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.944 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:16.945 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:16.945 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:16.945 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:16.945 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.945 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:16.945 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.946 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:16.946 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:16.946 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.946 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:16.946 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.947 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:16.947 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.947 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.947 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.947 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:16.947 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.947 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:16.948 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.948 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:16.948 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_lock_pp: /' 2023-10-30T20:25:16.948 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:16.948 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:16.949 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:16.949 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:16.949 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.949 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:16.949 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.949 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:16.950 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:16.950 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:16.950 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.950 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:16.951 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.951 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:16.951 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:16.951 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:16.951 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.951 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:16.952 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.952 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:16.952 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.952 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_pool 2023-10-30T20:25:16.952 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91557/exe 2023-10-30T20:25:16.953 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.953 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:16.953 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.953 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:16.953 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:16.953 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:16.954 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:16.954 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:16.954 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:16.954 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:16.954 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:16.955 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:16.955 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:16.955 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:16.955 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:16.955 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:16.955 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:16.956 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:16.956 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_lock.log 2023-10-30T20:25:16.956 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:16.956 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:16.956 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.957 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:16.957 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.957 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:16.957 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:16.957 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.958 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.958 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:16.958 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.958 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:16.959 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.959 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:16.959 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:16.959 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.959 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:16.959 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.960 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:16.960 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:16.960 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91610/exe 2023-10-30T20:25:16.960 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:16.960 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.960 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:16.960 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.961 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:16.961 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:16.961 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:16.961 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:16.961 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:16.962 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:16.962 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:16.962 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:16.962 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:16.962 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.963 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:16.963 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:16.963 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:16.963 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:16.963 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:16.963 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:16.964 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.964 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:16.964 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.964 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:16.964 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:16.964 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:16.965 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:16.965 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.965 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.965 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:16.965 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.965 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:16.965 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:16.966 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:16.966 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.966 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:16.966 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:16.966 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_pool 2023-10-30T20:25:16.966 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.967 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91705 2023-10-30T20:25:16.967 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_pool on pid 91705' 2023-10-30T20:25:16.967 INFO:tasks.workunit.client.0.smithi069.stdout:test api_pool on pid 91705 2023-10-30T20:25:16.972 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2023-10-30T20:25:16.973 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91705 2023-10-30T20:25:16.973 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91593/exe 2023-10-30T20:25:16.973 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:16.973 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:16.973 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_lock: /' 2023-10-30T20:25:16.974 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:16.974 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:16.974 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:16.974 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:16.974 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:16.974 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:16.975 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:16.975 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:16.975 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:16.975 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:16.975 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:16.976 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:16.976 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:16.976 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:16.976 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:16.976 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.977 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:16.977 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_misc.log 2023-10-30T20:25:16.977 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_lock 2023-10-30T20:25:16.977 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:16.977 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:16.978 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:16.978 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:16.978 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:16.978 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:16.978 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:16.979 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:16.979 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:16.979 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_misc 2023-10-30T20:25:16.979 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:16.979 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_misc_pp: /' 2023-10-30T20:25:16.979 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_misc_pp.log 2023-10-30T20:25:16.980 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_misc: /' 2023-10-30T20:25:16.980 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_misc_pp 2023-10-30T20:25:16.980 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_snapshots 2023-10-30T20:25:16.980 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_snapshots' 2023-10-30T20:25:16.980 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:16.981 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_snapshots 2023-10-30T20:25:16.981 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:16.981 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:16.981 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:16.981 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.982 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:16.982 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.982 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:16.982 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:16.982 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_snapshots 2023-10-30T20:25:16.983 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.983 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:16.983 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91737 2023-10-30T20:25:16.983 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.984 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:16.984 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.984 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.984 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.984 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:16.985 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.985 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:16.985 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.985 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_snapshots on pid 91737' 2023-10-30T20:25:16.986 INFO:tasks.workunit.client.0.smithi069.stdout:test api_snapshots on pid 91737 2023-10-30T20:25:16.992 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91737 2023-10-30T20:25:16.993 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:16.993 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:16.993 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2023-10-30T20:25:16.993 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_snapshots_pp 2023-10-30T20:25:16.993 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_snapshots_pp' 2023-10-30T20:25:16.994 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:16.994 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:16.994 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:16.994 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:16.994 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.995 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:16.995 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:16.995 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:16.995 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:16.995 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.996 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:16.996 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.996 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:16.996 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:16.996 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.996 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:16.997 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.997 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:16.997 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.997 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:16.997 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.997 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:16.998 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.998 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:16.998 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:16.998 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:16.998 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:16.999 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:16.999 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:16.999 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:16.999 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:16.999 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.000 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.000 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.000 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.000 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.001 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.001 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.001 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.001 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.001 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.002 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.002 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.002 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.002 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.002 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_snapshots_pp 2023-10-30T20:25:17.003 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.003 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.003 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.003 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.003 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.003 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.004 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.004 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.004 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.004 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.004 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.005 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.005 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.005 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.005 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.005 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.005 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.006 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.006 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.006 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.006 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.006 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_snapshots_pp 2023-10-30T20:25:17.006 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91763 2023-10-30T20:25:17.007 INFO:tasks.workunit.client.0.smithi069.stdout:test api_snapshots_pp on pid 91763 2023-10-30T20:25:17.016 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_snapshots_pp on pid 91763' 2023-10-30T20:25:17.016 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91763 2023-10-30T20:25:17.016 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:17.016 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.017 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.017 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.017 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.017 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.017 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.018 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.018 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.018 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.018 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.018 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.019 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.019 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.019 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_stat 2023-10-30T20:25:17.019 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.019 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_stat' 2023-10-30T20:25:17.019 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.020 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.020 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.020 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.020 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.020 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.020 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.021 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.021 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.021 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.021 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.021 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.022 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.022 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.022 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.022 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.022 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.023 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.023 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.023 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.023 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.023 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.024 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.024 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.024 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.024 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.024 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.024 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.025 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.025 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.025 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.025 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.025 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.025 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.026 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.026 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.026 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.026 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.026 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.026 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.027 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.028 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.028 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.028 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.028 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.028 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.028 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.029 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.029 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.029 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.029 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.029 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.029 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.030 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.030 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91648/exe 2023-10-30T20:25:17.030 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2023-10-30T20:25:17.030 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.030 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.030 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.031 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.031 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.031 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.031 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.031 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.031 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.032 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.032 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.032 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.032 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.032 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.033 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.033 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.033 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_tier_pp 2023-10-30T20:25:17.033 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.033 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.033 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.034 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.034 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.034 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.034 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.035 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.035 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.035 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.035 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.035 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_stat 2023-10-30T20:25:17.035 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.036 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.036 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_stat 2023-10-30T20:25:17.036 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2023-10-30T20:25:17.036 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.036 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.036 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.037 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.037 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.037 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.037 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.037 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91809 2023-10-30T20:25:17.037 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.038 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_stat on pid 91809' 2023-10-30T20:25:17.038 INFO:tasks.workunit.client.0.smithi069.stdout:test api_stat on pid 91809 2023-10-30T20:25:17.051 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91809 2023-10-30T20:25:17.051 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:17.052 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.052 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.052 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.052 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.052 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.053 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.053 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.053 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.053 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.053 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.053 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.054 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.054 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.054 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.054 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.054 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.055 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.055 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.055 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.055 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.055 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.057 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.057 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.057 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.058 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.058 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.058 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.058 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.059 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.059 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.059 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.059 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.059 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.059 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.060 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.060 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.060 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.060 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.060 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.060 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.061 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.061 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.061 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.061 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.061 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.061 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.062 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.062 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.062 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.062 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.062 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.063 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.063 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.063 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.063 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.063 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.063 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.064 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.064 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.064 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.064 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_tier_pp.log 2023-10-30T20:25:17.064 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_tier_pp: /' 2023-10-30T20:25:17.065 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.065 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.065 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.065 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.065 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.066 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.066 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.066 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.066 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.066 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.067 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.067 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.067 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.067 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.067 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.067 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.068 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.068 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.068 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.068 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.068 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.069 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.069 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.069 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.069 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.069 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.069 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.070 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.070 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.070 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.070 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.070 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.071 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.071 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.071 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.071 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.071 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.071 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.072 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.072 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.072 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.072 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.072 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.073 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.073 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.073 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.073 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.073 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.073 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.074 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.074 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.074 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.074 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.074 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.074 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.075 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.075 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.075 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.075 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.075 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.076 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.077 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.077 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91737/exe 2023-10-30T20:25:17.077 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.077 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91705/exe 2023-10-30T20:25:17.077 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.078 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.078 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.078 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.078 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.078 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.079 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.079 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.079 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.079 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.079 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.079 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.080 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.080 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.080 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.080 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.080 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.080 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.081 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.081 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.081 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.081 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.081 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.081 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.082 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.082 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.082 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.082 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.082 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.083 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.083 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.083 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.083 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.083 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.083 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.084 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.084 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.084 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.084 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.084 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.084 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_snapshots.log 2023-10-30T20:25:17.085 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_snapshots: /' 2023-10-30T20:25:17.085 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.085 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.085 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_snapshots 2023-10-30T20:25:17.085 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.086 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.086 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.086 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.086 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.086 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.086 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.087 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.087 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_stat_pp 2023-10-30T20:25:17.087 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_stat_pp' 2023-10-30T20:25:17.087 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_pool: /' 2023-10-30T20:25:17.087 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.087 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.088 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.088 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_stat_pp 2023-10-30T20:25:17.088 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.088 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.088 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.088 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.089 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.089 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.089 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_pool 2023-10-30T20:25:17.089 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.089 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.089 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.089 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.090 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.090 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.090 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.090 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.090 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.090 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.091 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.091 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.091 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.091 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.091 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.091 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.092 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.092 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_pool.log 2023-10-30T20:25:17.092 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2023-10-30T20:25:17.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] Global test environment set-up. 2023-10-30T20:25:17.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] 12 tests from AsioRados 2023-10-30T20:25:17.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.772+0000 7fa1b495b780 10 monclient: get_monmap_and_config 2023-10-30T20:25:17.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.772+0000 7fa1b495b780 10 monclient: build_initial_monmap 2023-10-30T20:25:17.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.772+0000 7fa1b495b780 10 monclient: monmap: 2023-10-30T20:25:17.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 0 2023-10-30T20:25:17.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:25:17.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 0.000000 2023-10-30T20:25:17.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 0.000000 2023-10-30T20:25:17.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 0 (unknown) 2023-10-30T20:25:17.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 1 2023-10-30T20:25:17.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:25:17.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:25:17.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:25:17.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 1 Processor -- start 2023-10-30T20:25:17.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 1 -- start start 2023-10-30T20:25:17.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 10 monclient: init 2023-10-30T20:25:17.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 10 monclient: _reopen_session rank -1 2023-10-30T20:25:17.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T20:25:17.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d109d9f330 0x55d109e2b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:25:17.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 10 monclient(hunting): picked mon.noname-b con 0x55d109d9f330 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:25:17.097 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d109da0a30 0x55d109e31680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:25:17.097 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 10 monclient(hunting): picked mon.noname-a con 0x55d109da0a30 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:25:17.097 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d109e32e50 0x55d109da5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:25:17.097 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 10 monclient(hunting): picked mon.noname-c con 0x55d109e32e50 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:25:17.097 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 10 monclient(hunting): start opening mon connection 2023-10-30T20:25:17.097 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d109cd5220 con 0x55d109da0a30 2023-10-30T20:25:17.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 10 monclient(hunting): start opening mon connection 2023-10-30T20:25:17.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d109ca9d20 con 0x55d109d9f330 2023-10-30T20:25:17.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 10 monclient(hunting): start opening mon connection 2023-10-30T20:25:17.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.773+0000 7fa1b495b780 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d109ca7160 con 0x55d109e32e50 2023-10-30T20:25:17.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.774+0000 7fa1b495b780 10 monclient(hunting): _renew_subs 2023-10-30T20:25:17.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.774+0000 7fa1b495b780 10 monclient(hunting): authenticate will time out at 1640.063120s 2023-10-30T20:25:17.099 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_stat_pp 2023-10-30T20:25:17.099 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.100 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.100 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.100 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91882 2023-10-30T20:25:17.100 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.100 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_stat_pp on pid 91882' 2023-10-30T20:25:17.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: test api_stat_pp on pid 91882 2023-10-30T20:25:17.102 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91882 2023-10-30T20:25:17.103 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.103 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.103 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:17.103 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.104 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.104 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.104 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.104 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.104 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.104 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.105 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.105 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.105 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.105 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.105 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.106 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.106 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.106 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.106 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.106 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2023-10-30T20:25:17.106 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_watch_notify 2023-10-30T20:25:17.107 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.107 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.107 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_watch_notify' 2023-10-30T20:25:17.107 INFO:tasks.workunit.client.0.smithi069.stdout:2023-10-30T20:25:16.775+0000 7fa1b2c60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d109e32e50 0x55d109da5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:25:17.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.775+0000 7fa1b2c60640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d109e32e50 0x55d109da5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:43964/0 (socket says 172.21.15.69:43964) 2023-10-30T20:25:17.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.775+0000 7fa1b2c60640 1 -- 172.21.15.69:0/83510205 learned_addr learned my addr 172.21.15.69:0/83510205 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:25:17.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.775+0000 7fa1b2c60640 10 monclient(hunting): get_auth_request con 0x55d109e32e50 auth_method 0 2023-10-30T20:25:17.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.775+0000 7fa1b2c60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:25:17.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.775+0000 7fa1b2c60640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:25:17.109 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.775+0000 7fa1b2c60640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:25:17.109 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.775+0000 7fa1b1c5e640 1 --2- 172.21.15.69:0/83510205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d109da0a30 0x55d109e31680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:25:17.109 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.776+0000 7fa1b1c5e640 10 monclient(hunting): get_auth_request con 0x55d109da0a30 auth_method 0 2023-10-30T20:25:17.109 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.776+0000 7fa1b1c5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:25:17.109 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.776+0000 7fa1b1c5e640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:25:17.109 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.776+0000 7fa1b1c5e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:25:17.110 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.776+0000 7fa1b1c5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:25:17.110 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.776+0000 7fa1b1c5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:25:17.110 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.776+0000 7fa1b1c5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:25:17.110 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.779+0000 7fa1b2c60640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:25:17.110 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.779+0000 7fa1b2c60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:25:17.111 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.779+0000 7fa1b2c60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:25:17.111 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.779+0000 7fa1b1c5e640 10 monclient(hunting): handle_auth_done global_id 4377 payload 274 2023-10-30T20:25:17.111 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.779+0000 7fa1b1c5e640 1 -- 172.21.15.69:0/83510205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d109e32e50 msgr2=0x55d109da5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:25:17.111 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.779+0000 7fa1b1c5e640 1 --2- 172.21.15.69:0/83510205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d109e32e50 0x55d109da5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:25:17.111 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.779+0000 7fa1b1c5e640 1 -- 172.21.15.69:0/83510205 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d109d9f330 msgr2=0x55d109e2b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:25:17.116 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.116 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_watch_notify 2023-10-30T20:25:17.116 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.116 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.116 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.117 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.117 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.118 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.118 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.118 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.118 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.118 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.118 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.119 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.119 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.119 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.119 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.119 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.119 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.120 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.120 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.120 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.120 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.120 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.120 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.121 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.121 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.121 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.121 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.121 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.122 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.123 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.123 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.123 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.124 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.124 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.124 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.124 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.124 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.125 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.125 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.125 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.125 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.125 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.125 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_watch_notify 2023-10-30T20:25:17.126 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91895 2023-10-30T20:25:17.126 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_watch_notify on pid 91895' 2023-10-30T20:25:17.126 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.779+0000 7fa1b1c5e640 1 --2- 172.21.15.69:0/83510205 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d109d9f330 0x55d109e2b4f0 unknown :-1 s=BANNER_CONNECtest api_watch_notify on pid 91895 2023-10-30T20:25:17.133 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91895 2023-10-30T20:25:17.134 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:17.134 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.134 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_watch_notify_pp 2023-10-30T20:25:17.134 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_watch_notify_pp' 2023-10-30T20:25:17.135 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91763/exe 2023-10-30T20:25:17.135 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.135 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.135 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.135 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.135 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.136 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.136 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.136 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.136 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.136 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.136 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.137 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.139 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2023-10-30T20:25:17.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.140 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.140 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.141 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.141 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.141 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.141 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.141 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.141 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.142 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.142 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.142 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.142 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91809/exe 2023-10-30T20:25:17.142 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.143 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.143 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.143 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.143 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.143 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.143 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.144 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.144 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.144 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.144 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.144 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.144 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.145 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.145 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.145 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.145 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.145 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.145 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.146 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.146 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.146 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2023-10-30T20:25:17.146 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.147 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.147 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_snapshots_pp 2023-10-30T20:25:17.147 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.147 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.147 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.147 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.148 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.149 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.150 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.150 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.150 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.150 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.150 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.151 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.151 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.151 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.151 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.151 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.151 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.152 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.152 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.152 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.152 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.152 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.152 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.153 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.153 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.153 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.153 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.153 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.153 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.154 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.154 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.154 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.154 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.154 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_watch_notify_pp 2023-10-30T20:25:17.155 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.155 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.155 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.155 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.155 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_snapshots_pp: /' 2023-10-30T20:25:17.156 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.156 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.156 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.156 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.156 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.156 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.157 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_stat 2023-10-30T20:25:17.157 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_stat.log 2023-10-30T20:25:17.157 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_stat: /' 2023-10-30T20:25:17.157 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_watch_notify_pp 2023-10-30T20:25:17.157 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91946 2023-10-30T20:25:17.157 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_watch_notify_pp on pid 91946' 2023-10-30T20:25:17.158 INFO:tasks.workunit.client.0.smithi069.stdout:test api_watch_notify_pp on pid 91946 2023-10-30T20:25:17.163 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91946 2023-10-30T20:25:17.163 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:17.164 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.164 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2023-10-30T20:25:17.164 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.164 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.164 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.165 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.165 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.166 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cmd 2023-10-30T20:25:17.166 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cmd' 2023-10-30T20:25:17.166 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.166 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.166 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.167 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.167 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.167 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.167 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cmd 2023-10-30T20:25:17.167 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.167 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.168 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.168 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.168 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.168 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.168 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.169 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.169 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.169 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.169 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.169 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.170 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.170 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.170 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.170 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.171 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.171 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.171 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.171 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.171 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.171 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.172 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.172 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.172 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.172 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.172 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.173 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.173 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.173 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.173 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.173 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.173 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.174 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.174 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.174 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.174 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.174 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.175 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.175 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.175 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.175 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.175 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.176 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.176 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.176 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.176 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.176 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.176 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cmd 2023-10-30T20:25:17.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.178 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.178 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91970 2023-10-30T20:25:17.178 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.178 INFO:tasks.workunit.client.0.smithi069.stdout:TING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:25:17.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.779+0000 7fa1b1c5e640 10 monclient: _finish_hunting 0 2023-10-30T20:25:17.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.779+0000 7fa1b1c5e640 1 monclient: found mon.noname-a 2023-10-30T20:25:17.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.779+0000 7fa1b1c5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:25:17.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.779+0000 7fa1b1c5e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:25:17.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.779+0000 7fa1b1c5e640 1 -- 172.21.15.69:0/83510205 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d109e31bc0 con 0x55d109da0a30 2023-10-30T20:25:17.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.790+0000 7fa1b1c5e640 1 --2- 172.21.15.69:0/83510205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d109da0a30 0x55d109e31680 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8055000 tx=0x7fa1a8050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ccd93e1661a7910 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:25:17.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b0c5c640 1 -- 172.21.15.69:0/83510205 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8056640 con 0x55d109da0a30 2023-10-30T20:25:17.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b0c5c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:25:17.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b0c5c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:25:17.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b0c5c640 10 monclient: dump: 2023-10-30T20:25:17.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 1 2023-10-30T20:25:17.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:25:17.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:25:17.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:25:17.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T20:25:17.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 3 2023-10-30T20:25:17.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:25:17.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:25:17.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:25:17.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b0c5c640 10 monclient: _finish_auth 0 2023-10-30T20:25:17.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b0c5c640 10 monclient: _check_auth_tickets 2023-10-30T20:25:17.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b0c5c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:17.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b0c5c640 1 -- 172.21.15.69:0/83510205 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a800f280 con 0x55d109da0a30 2023-10-30T20:25:17.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b0c5c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:25:17.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b0c5c640 1 -- 172.21.15.69:0/83510205 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8056b70 con 0x55d109da0a30 2023-10-30T20:25:17.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b0c5c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:25:17.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b0c5c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:25:17.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b0c5c640 10 monclient: dump: 2023-10-30T20:25:17.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 1 2023-10-30T20:25:17.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:25:17.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:25:17.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:25:17.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T20:25:17.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 3 2023-10-30T20:25:17.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:25:17.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:25:17.187 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.187 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.187 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.188 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.188 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.188 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.188 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.188 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.189 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.189 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.189 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.189 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.190 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.190 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.190 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.190 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.191 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.191 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.191 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.191 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.191 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.192 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.192 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cmd on pid 91970' 2023-10-30T20:25:17.192 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.192 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.192 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.193 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2:test api_cmd on pid 91970 2023-10-30T20:25:17.196 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91970 2023-10-30T20:25:17.196 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:17.197 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.197 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.197 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2023-10-30T20:25:17.197 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.197 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.198 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.198 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.198 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.198 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cmd_pp 2023-10-30T20:25:17.198 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.199 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cmd_pp' 2023-10-30T20:25:17.199 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.199 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.199 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.199 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.200 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.200 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.200 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.200 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.200 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.200 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.201 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.201 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.201 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.201 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.201 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.201 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.202 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.202 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.202 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.202 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.202 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.203 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.203 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.203 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cmd_pp 2023-10-30T20:25:17.203 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.203 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.203 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.204 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.204 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.204 INFO:tasks.workunit.client.0.smithi069.stdout: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:25:17.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b495b780 5 monclient: authenticate success, global_id 4377 2023-10-30T20:25:17.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b495b780 10 monclient: get_monmap_and_config success 2023-10-30T20:25:17.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.825+0000 7fa1b495b780 4 set_mon_vals no callback set 2023-10-30T20:25:17.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.827+0000 7fa1b495b780 10 monclient: shutdown 2023-10-30T20:25:17.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.827+0000 7fa1b495b780 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:25:17.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.827+0000 7fa1b495b780 1 -- 172.21.15.69:0/83510205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d109da0a30 msgr2=0x55d109e31680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:25:17.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.827+0000 7fa1b495b780 1 --2- 172.21.15.69:0/83510205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d109da0a30 0x55d109e31680 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8055000 tx=0x7fa1a8050000 comp rx=0 tx=0).stop 2023-10-30T20:25:17.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.827+0000 7fa1b495b780 1 -- 172.21.15.69:0/83510205 shutdown_connections 2023-10-30T20:25:17.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.827+0000 7fa1b495b780 1 --2- 172.21.15.69:0/83510205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d109e32e50 0x55d109da5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:25:17.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.827+0000 7fa1b495b780 1 --2- 172.21.15.69:0/83510205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d109da0a30 0x55d109e31680 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:25:17.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.827+0000 7fa1b495b780 1 --2- 172.21.15.69:0/83510205 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d109d9f330 0x55d109e2b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:25:17.207 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.827+0000 7fa1b495b780 1 -- 172.21.15.69:0/83510205 >> 172.21.15.69:0/83510205 conn(0x55d109da5890 msgr2=0x55d109da4930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:25:17.207 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.827+0000 7fa1b495b780 1 -- 172.21.15.69:0/83510205 shutdown_connections 2023-10-30T20:25:17.207 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.827+0000 7fa1b495b780 1 -- 172.21.15.69:0/83510205 wait complete. 2023-10-30T20:25:17.207 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.827+0000 7fa1b495b780 10 monclient: build_initial_monmap 2023-10-30T20:25:17.207 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.827+0000 7fa1b495b780 10 monclient: monmap: 2023-10-30T20:25:17.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 0 2023-10-30T20:25:17.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:25:17.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 0.000000 2023-10-30T20:25:17.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 0.000000 2023-10-30T20:25:17.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 0 (unknown) 2023-10-30T20:25:17.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 1 2023-10-30T20:25:17.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:25:17.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:25:17.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:25:17.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.828+0000 7fa1b495b780 1 librados: starting msgr at 2023-10-30T20:25:17.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.828+0000 7fa1b495b780 1 librados: starting objecter 2023-10-30T20:25:17.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.828+0000 7fa1b495b780 1 Processor -- start 2023-10-30T20:25:17.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.828+0000 7fa1b495b780 1 -- start start 2023-10-30T20:25:17.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.828+0000 7fa1b495b780 1 librados: setting wanted keys 2023-10-30T20:25:17.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.828+0000 7fa1b495b780 1 librados: calling monclient init 2023-10-30T20:25:17.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.828+0000 7fa1b495b780 10 monclient: init 2023-10-30T20:25:17.214 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.214 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.214 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.215 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.215 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.215 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.215 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.215 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.216 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.216 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.216 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.216 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.216 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.216 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.217 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.217 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.217 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.217 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.217 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.218 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.218 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.218 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.218 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.218 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.218 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.219 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.219 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.219 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.219 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91895/exe 2023-10-30T20:25:17.219 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.220 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.220 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.220 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.220 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.220 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.220 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.221 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.221 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.221 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.221 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.221 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cmd_pp 2023-10-30T20:25:17.221 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.222 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.222 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91996 2023-10-30T20:25:17.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 10 monclietest api_cmd_pp on pid 91996 2023-10-30T20:25:17.228 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cmd_pp on pid 91996' 2023-10-30T20:25:17.228 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91996 2023-10-30T20:25:17.229 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:17.229 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.229 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.229 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.229 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.230 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.230 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2023-10-30T20:25:17.231 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.231 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91882/exe 2023-10-30T20:25:17.231 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.231 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.231 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_service 2023-10-30T20:25:17.232 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.232 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.232 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.232 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.232 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_service' 2023-10-30T20:25:17.232 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.233 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.233 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.233 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.233 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.233 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.234 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.234 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.234 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.234 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.234 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.235 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.235 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.235 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.235 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.235 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.235 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.236 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.236 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.236 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.236 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.236 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.236 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.237 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.237 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.237 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.237 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.237 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.237 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.238 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.238 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.238 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.238 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.238 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.239 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.239 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.239 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.239 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.239 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.239 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.240 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.240 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.240 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.240 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.240 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.240 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.241 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.241 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.241 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.241 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_watch_notify 2023-10-30T20:25:17.241 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.242 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.242 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.242 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_watch_notify.log 2023-10-30T20:25:17.242 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.242 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.243 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_watch_notify: /' 2023-10-30T20:25:17.243 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.243 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.243 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.243 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.243 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.244 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.244 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.244 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_stat_pp.log 2023-10-30T20:25:17.244 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_service 2023-10-30T20:25:17.244 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_service 2023-10-30T20:25:17.245 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92020 2023-10-30T20:25:17.245 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_service on pid 92020' 2023-10-30T20:25:17.245 INFO:tasks.workunit.client.0.smithi069.stdout:test api_service on pid 92020 2023-10-30T20:25:17.251 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92020 2023-10-30T20:25:17.251 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:17.252 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.252 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2023-10-30T20:25:17.252 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.252 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.252 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.253 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.253 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.253 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.253 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.253 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.254 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.254 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.254 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.254 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.254 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.254 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.255 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.255 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.255 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.255 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.255 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.256 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.256 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.256 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.256 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.256 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.256 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.257 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.257 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_service_pp 2023-10-30T20:25:17.257 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_service_pp' 2023-10-30T20:25:17.257 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_stat_pp 2023-10-30T20:25:17.257 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_stat_pp: /' 2023-10-30T20:25:17.258 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.258 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.258 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.258 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.258 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.259 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.260 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.260 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.260 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.260 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.260 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_service_pp 2023-10-30T20:25:17.261 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.261 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.261 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.261 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.261 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.261 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.262 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.262 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.262 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.262 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.262 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.263 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.264 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.264 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.264 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.264 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.264 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.264 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.265 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.265 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.265 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.265 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.265 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.265 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.266 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.266 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.266 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.266 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.266 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.267 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_service_pp 2023-10-30T20:25:17.267 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92046 2023-10-30T20:25:17.267 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_service_pp on pid 92046' 2023-10-30T20:25:17.267 INFO:tasks.workunit.client.0.smithi069.stdout:test api_service_pp on pid 92046 2023-10-30T20:25:17.275 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.275 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.275 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92046 2023-10-30T20:25:17.276 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:17.276 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.276 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.276 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.276 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.276 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.277 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.277 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.277 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.277 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_c_write_operations 2023-10-30T20:25:17.277 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_c_write_operations' 2023-10-30T20:25:17.277 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.278 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2023-10-30T20:25:17.278 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.278 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.278 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.278 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.279 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.279 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.279 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.279 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.279 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.279 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.280 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.280 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.280 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.280 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.280 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.280 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.281 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.281 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.281 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.281 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.281 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.281 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.282 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.282 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.282 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.282 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.282 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.283 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_c_write_operations 2023-10-30T20:25:17.283 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.283 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.283 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.283 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.283 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.284 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.284 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.284 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.284 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.284 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.284 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.285 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.285 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.285 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.285 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.285 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.285 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.286 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.286 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.286 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.286 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.286 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.287 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.287 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.288 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.288 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.289 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.289 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.289 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.289 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.289 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.289 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.290 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.290 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.290 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.290 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.290 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.291 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.291 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.291 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.291 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.291 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.291 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.292 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.292 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.292 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_c_write_operations 2023-10-30T20:25:17.292 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.292 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.292 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.293 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.293 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.293 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.293 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92064 2023-10-30T20:25:17.293 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.294 INFO:tasks.workunit.client.0.smithi069.stdout:test api_c_write_operations on pid 92064 2023-10-30T20:25:17.299 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_c_write_operations on pid 92064' 2023-10-30T20:25:17.300 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92064 2023-10-30T20:25:17.300 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:17.300 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.300 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.300 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.301 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.301 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.301 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.301 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.301 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.301 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.302 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.302 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.302 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.302 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.302 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.303 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.303 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.303 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.303 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.303 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.304 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.304 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.304 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.304 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.304 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.305 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_c_read_operations 2023-10-30T20:25:17.305 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_c_read_operations' 2023-10-30T20:25:17.305 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.305 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.305 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.305 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.306 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.306 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.306 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.306 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.307 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.307 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.307 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91946/exe 2023-10-30T20:25:17.307 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.307 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.308 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.308 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.308 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.308 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.308 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.309 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.309 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.309 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.309 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.309 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.309 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.310 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.310 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.310 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.310 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.310 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.311 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.311 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.311 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.311 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.311 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.311 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.312 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.312 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.312 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.312 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.312 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.313 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.313 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91996/exe 2023-10-30T20:25:17.314 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91970/exe 2023-10-30T20:25:17.314 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.314 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.314 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.314 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.314 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.315 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.315 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.315 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.315 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.315 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.316 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.316 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.316 INFO:tasks.workunit.client.0.smithi069.stdout:nt: _reopen_session rank -1 2023-10-30T20:25:17.316 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T20:25:17.317 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d109d9f330 0x55d109e39870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:25:17.317 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 10 monclient(hunting): picked mon.noname-a con 0x55d109d9f330 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:25:17.317 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d109da0a30 0x55d109e2d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:25:17.317 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 10 monclient(hunting): picked mon.noname-c con 0x55d109da0a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:25:17.317 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d109e32e50 0x55d109d66c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:25:17.318 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 10 monclient(hunting): picked mon.noname-b con 0x55d109e32e50 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:25:17.318 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 10 monclient(hunting): start opening mon connection 2023-10-30T20:25:17.318 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d109e2fab0 con 0x55d109d9f330 2023-10-30T20:25:17.318 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 10 monclient(hunting): start opening mon connection 2023-10-30T20:25:17.318 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d109e32a00 con 0x55d109e32e50 2023-10-30T20:25:17.319 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 10 monclient(hunting): start opening mon connection 2023-10-30T20:25:17.319 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d109d67540 con 0x55d109da0a30 2023-10-30T20:25:17.319 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 10 monclient(hunting): _renew_subs 2023-10-30T20:25:17.319 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.829+0000 7fa1b495b780 10 monclient(hunting): authenticate will time out at 1640.118109s 2023-10-30T20:25:17.319 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.862+0000 7fa1b2c60640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d109e32e50 0x55d109d66c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:25:17.319 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.864+0000 7fa1b2c60640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d109e32e50 0x55d109d66c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:39346/0 (socket says 172.21.15.69:39346) 2023-10-30T20:25:17.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.864+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 learned_addr learned my addr 172.21.15.69:0/1144027014 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:25:17.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.864+0000 7fa1b2c60640 10 monclient(hunting): get_auth_request con 0x55d109e32e50 auth_method 0 2023-10-30T20:25:17.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.864+0000 7fa1b2c60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:25:17.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.864+0000 7fa1b2c60640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:25:17.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.864+0000 7fa1b2c60640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:25:17.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2023-10-30T20:25:17.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2023-10-30T20:25:17.331 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.331 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.332 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.332 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.332 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.332 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.332 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.334 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.334 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.334 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.334 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.334 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.335 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.335 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.335 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.335 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.335 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.335 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.336 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.336 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.336 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.336 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.336 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.336 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.337 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.337 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.337 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.337 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.337 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.337 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.338 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.338 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.338 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.338 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.338 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.338 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.339 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.339 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.339 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.339 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.339 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_watch_notify_pp 2023-10-30T20:25:17.339 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.340 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.340 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.340 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.340 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.340 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.340 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.341 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.341 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2023-10-30T20:25:17.341 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.341 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.341 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.341 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.342 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.342 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.342 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.342 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.342 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.343 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cmd_pp 2023-10-30T20:25:17.343 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.343 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cmd_pp: /' 2023-10-30T20:25:17.343 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.343 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.344 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.344 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.344 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.344 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.344 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.344 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.345 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.345 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.345 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.345 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.345 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.346 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.346 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.346 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.346 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.346 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.347 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.347 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.347 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.347 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.347 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.347 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.348 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.348 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.348 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.348 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.348 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.348 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.349 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.349 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.349 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.349 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.349 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.350 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.350 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.350 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.350 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.350 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.350 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.351 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.351 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.351 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.351 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.351 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.351 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.352 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.352 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.352 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.352 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.352 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.353 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.353 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.353 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.353 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.353 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.353 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.354 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.354 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.354 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.354 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.354 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.354 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.355 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.355 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.355 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_c_read_operations 2023-10-30T20:25:17.355 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cmd 2023-10-30T20:25:17.355 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.356 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.356 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.356 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.356 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.356 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.356 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.357 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.357 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.357 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.357 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.357 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.357 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.358 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.358 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.358 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.358 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.358 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.359 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.359 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.359 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cmd.log 2023-10-30T20:25:17.359 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cmd: /' 2023-10-30T20:25:17.359 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.360 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2023-10-30T20:25:17.360 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_c_read_operations 2023-10-30T20:25:17.360 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92114 2023-10-30T20:25:17.360 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_c_read_operations on pid 92114' 2023-10-30T20:25:17.360 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.865+0000 7fa1b1c5e640 1 --2- 172.21.15.69:0/1144027014 >> [v2:172.test api_c_read_operations on pid 92114 2023-10-30T20:25:17.365 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92114 2023-10-30T20:25:17.365 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:17.365 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.365 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.365 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.366 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.366 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2023-10-30T20:25:17.366 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.366 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.366 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.366 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.367 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.367 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.367 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.367 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.367 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.367 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.368 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.368 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.368 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.368 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.368 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.369 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.369 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.369 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.369 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.369 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.369 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.370 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cls_remote_reads 2023-10-30T20:25:17.370 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cls_remote_reads' 2023-10-30T20:25:17.370 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.370 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.370 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.371 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.371 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.371 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.371 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.371 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.372 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.372 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.372 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.372 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.372 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.372 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.373 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.373 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.373 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.373 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.373 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cls_remote_reads 2023-10-30T20:25:17.373 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cls_remote_reads 2023-10-30T20:25:17.374 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.374 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92136 2023-10-30T20:25:17.374 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cls_remote_reads on pid 92136' 2023-10-30T20:25:17.374 INFO:tasks.workunit.client.0.smithi069.stdout:test api_cls_remote_reads on pid 92136 2023-10-30T20:25:17.378 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92136 2023-10-30T20:25:17.378 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:17.378 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.378 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s list_parallel 2023-10-30T20:25:17.379 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' list_parallel' 2023-10-30T20:25:17.379 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.379 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.379 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.379 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.379 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.380 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo list_parallel 2023-10-30T20:25:17.380 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.380 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.380 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.380 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.381 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.381 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.381 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.381 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.381 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.381 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.382 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2023-10-30T20:25:17.382 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.382 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.382 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.382 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.383 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.383 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.384 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.384 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=list_parallel 2023-10-30T20:25:17.384 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.384 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92149 2023-10-30T20:25:17.384 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test list_parallel on pid 92149' 2023-10-30T20:25:17.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.385 INFO:tasks.workunit.client.0.smithi069.stdout:test list_parallel on pid 92149 2023-10-30T20:25:17.401 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92149 2023-10-30T20:25:17.402 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:17.402 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.402 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.402 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.403 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.403 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.403 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.403 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.404 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.404 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.404 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.404 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.404 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.405 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92020/exe 2023-10-30T20:25:17.405 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.406 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.406 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.406 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.407 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.407 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.407 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.407 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.408 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.408 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.408 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.408 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.408 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.409 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.409 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.409 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.409 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.409 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.409 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.410 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.410 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.410 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.410 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.410 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.410 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.411 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.411 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.411 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.411 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.412 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.412 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.412 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.412 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.412 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.413 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.414 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.414 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.414 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.414 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.414 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.415 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.415 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.415 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92046/exe 2023-10-30T20:25:17.415 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.415 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.416 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.416 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.416 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.416 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.416 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.417 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.417 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.417 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.417 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.417 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.417 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.418 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.418 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.418 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.418 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.418 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.419 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.419 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2023-10-30T20:25:17.419 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.419 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92064/exe 2023-10-30T20:25:17.419 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.419 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.420 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.420 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.420 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.420 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.420 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.421 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.421 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.421 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.421 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.421 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.422 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.422 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.422 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.422 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.422 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.423 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_service_pp 2023-10-30T20:25:17.423 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_service_pp: /' 2023-10-30T20:25:17.423 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.423 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.423 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.423 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.424 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.424 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.424 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.424 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.424 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_service_pp.log 2023-10-30T20:25:17.425 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.425 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.425 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.426 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.426 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.427 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.427 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.427 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.427 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.428 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.428 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.428 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.428 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.428 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.428 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.429 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.429 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.429 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.429 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.429 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.429 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.430 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.430 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.430 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_service 2023-10-30T20:25:17.430 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_service.log 2023-10-30T20:25:17.430 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.431 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.431 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s open_pools_parallel 2023-10-30T20:25:17.431 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.431 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.431 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.431 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.432 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.432 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.432 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.432 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2023-10-30T20:25:17.432 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.433 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.433 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.433 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.433 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.433 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.433 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.434 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.434 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.434 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.434 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.434 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' open_pools_parallel' 2023-10-30T20:25:17.435 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_service: /' 2023-10-30T20:25:17.435 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.435 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.435 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.435 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_c_write_operations 2023-10-30T20:25:17.436 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.436 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.436 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_c_write_operations: /' 2023-10-30T20:25:17.436 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.436 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.437 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.438 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.438 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.438 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.438 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.438 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.439 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.439 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.439 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.439 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.439 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.440 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.440 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo open_pools_parallel 2023-10-30T20:25:17.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.441 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.441 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.441 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.441 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.442 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.442 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.442 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.442 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.442 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.443 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.443 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.443 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.443 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.443 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.443 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.444 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.444 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=open_pools_parallel 2023-10-30T20:25:17.444 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.444 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.444 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.445 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92214 2023-10-30T20:25:17.445 INFO:tasks.workunit.client.0.smithi069.stdout:test open_pools_parallel on pid 92214 2023-10-30T20:25:17.453 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test open_pools_parallel on pid 92214' 2023-10-30T20:25:17.454 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92214 2023-10-30T20:25:17.454 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T20:25:17.454 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.454 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.454 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.455 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.455 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.455 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.455 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.455 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.456 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.456 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.456 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.456 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.456 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.456 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.457 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2023-10-30T20:25:17.457 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s delete_pools_parallel 2023-10-30T20:25:17.457 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' delete_pools_parallel' 2023-10-30T20:25:17.457 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.457 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.458 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.458 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.458 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.458 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.458 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo delete_pools_parallel 2023-10-30T20:25:17.458 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.459 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.459 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.459 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.459 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.459 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.460 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.460 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.460 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.460 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.460 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.460 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.461 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.461 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.461 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.461 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.461 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.461 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.462 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.462 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.462 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.462 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.462 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.465 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.465 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.465 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.466 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.466 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.466 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.466 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.466 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.467 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.467 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.467 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.467 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.467 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.467 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.468 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.468 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.468 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.468 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.468 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.468 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.469 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.469 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.469 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.469 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.469 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.469 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.470 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.471 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.471 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.471 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.472 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.472 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.472 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.473 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.473 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.473 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.473 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.473 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-10-30T20:25:17.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.475 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.475 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.475 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.476 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.476 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.476 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.476 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.476 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.477 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.477 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.477 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.477 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.477 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.477 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.478 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.478 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.478 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=delete_pools_parallel 2023-10-30T20:25:17.478 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.478 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.479 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92248 2023-10-30T20:25:17.479 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.479 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test delete_pools_parallel on pid 92248' 2023-10-30T20:25:17.479 INFO:tasks.workunit.client.0.smithi069.stdout:test delete_pools_parallel on pid 92248 2023-10-30T20:25:17.487 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92248 2023-10-30T20:25:17.487 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.488 INFO:tasks.workunit.client.0.smithi069.stderr:+ ret=0 2023-10-30T20:25:17.488 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.488 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.488 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T20:25:17.488 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.488 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.489 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:25:17.489 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91648 2023-10-30T20:25:17.489 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.489 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91648 2023-10-30T20:25:17.489 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.490 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.490 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.490 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.490 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.490 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.491 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.492 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.492 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.492 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.492 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92149/exe 2023-10-30T20:25:17.492 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.492 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.493 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.493 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.493 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.493 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.493 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.493 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.494 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.494 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.494 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.494 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.494 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.494 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.495 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.495 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.495 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.495 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.495 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.496 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.497 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.497 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.497 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.497 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.497 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.497 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.498 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.498 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.498 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.498 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.498 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.498 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.499 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.499 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.499 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.499 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.499 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.500 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.500 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.500 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.500 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.500 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.501 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.501 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_list_parallel 2023-10-30T20:25:17.501 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_list_parallel.log 2023-10-30T20:25:17.501 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92136/exe 2023-10-30T20:25:17.501 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.501 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.502 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.503 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.503 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.503 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.503 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.503 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.504 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.504 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.504 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.504 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.504 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.504 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.505 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.505 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.505 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cls_remote_reads 2023-10-30T20:25:17.505 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.505 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.506 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.506 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.506 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.506 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.506 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.507 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.507 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.507 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2023-10-30T20:25:17.507 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.507 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.507 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.508 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.508 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.508 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.508 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.508 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.509 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T20:25:17.509 INFO:tasks.workunit.client.0.smithi069.stdout:21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d109da0a30 0x55d109e2d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:25:17.509 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.865+0000 7fa1b1c5e640 10 monclient(hunting): get_auth_request con 0x55d109da0a30 auth_method 0 2023-10-30T20:25:17.509 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.865+0000 7fa1b1c5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:25:17.510 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.865+0000 7fa1b1c5e640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:25:17.510 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.865+0000 7fa1b1c5e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:25:17.510 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.866+0000 7fa1b1c5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:25:17.510 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.866+0000 7fa1b1c5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:25:17.510 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.866+0000 7fa1b1c5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:25:17.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.871+0000 7fa1b2c60640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:25:17.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.871+0000 7fa1b2c60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:25:17.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.871+0000 7fa1b2c60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:25:17.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.876+0000 7fa1b1c5e640 10 monclient(hunting): handle_auth_done global_id 4367 payload 824 2023-10-30T20:25:17.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.876+0000 7fa1b1c5e640 1 -- 172.21.15.69:0/1144027014 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d109e32e50 msgr2=0x55d109d66c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:25:17.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.876+0000 7fa1b1c5e640 1 --2- 172.21.15.69:0/1144027014 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d109e32e50 0x55d109d66c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:25:17.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.876+0000 7fa1b1c5e640 1 -- 172.21.15.69:0/1144027014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d109d9f330 msgr2=0x55d109e39870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:25:17.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.876+0000 7fa1b1c5e640 1 --2- 172.21.15.69:0/1144027014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d109d9f330 0x55d109e39870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:25:17.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.876+0000 7fa1b1c5e640 10 monclient: _finish_hunting 0 2023-10-30T20:25:17.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.876+0000 7fa1b1c5e640 1 monclient: found mon.noname-c 2023-10-30T20:25:17.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.876+0000 7fa1b1c5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:25:17.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.876+0000 7fa1b1c5e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:25:17.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.876+0000 7fa1b1c5e640 1 -- 172.21.15.69:0/1144027014 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d109d66570 con 0x55d109da0a30 2023-10-30T20:25:17.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.900+0000 7fa1b1c5e640 1 --2- 172.21.15.69:0/1144027014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d109da0a30 0x55d109e2d820 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55d109da06e0 tx=0x7fa1a8002fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:25:17.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.900+0000 7fa1a37fe640 10 client.?.objecter ms_handle_connect 0x55d109da0a30 2023-10-30T20:25:17.514 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.900+0000 7fa1a37fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:25:17.514 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.955+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8056d40 con 0x55d109da0a30 2023-10-30T20:25:17.514 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.955+0000 7fa1a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:25:17.514 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.955+0000 7fa1a37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:25:17.514 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.955+0000 7fa1a37fe640 10 monclient: dump: 2023-10-30T20:25:17.514 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 1 2023-10-30T20:25:17.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:25:17.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:25:17.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:25:17.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T20:25:17.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 3 2023-10-30T20:25:17.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:25:17.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:25:17.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:25:17.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.955+0000 7fa1a37fe640 10 monclient: _finish_auth 0 2023-10-30T20:25:17.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.955+0000 7fa1a37fe640 10 monclient: _check_auth_tickets 2023-10-30T20:25:17.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.955+0000 7fa1a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:17.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.955+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a805f410 con 0x55d109da0a30 2023-10-30T20:25:17.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.955+0000 7fa1a37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:25:17.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.955+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a805f7b0 con 0x55d109da0a30 2023-10-30T20:25:17.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.955+0000 7fa1a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:25:17.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.955+0000 7fa1a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:25:17.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.955+0000 7fa1a37fe640 10 monclient: dump: 2023-10-30T20:25:17.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 1 2023-10-30T20:25:17.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:25:17.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:25:17.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:25:17.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T20:25:17.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 3 2023-10-30T20:25:17.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:25:17.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:25:17.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:25:17.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.955+0000 7fa1b0c5c640 4 set_mon_vals no callback set 2023-10-30T20:25:17.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.956+0000 7fa1b495b780 5 monclient: authenticate success, global_id 4367 2023-10-30T20:25:17.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.956+0000 7fa1b495b780 10 monclient: _renew_subs 2023-10-30T20:25:17.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.956+0000 7fa1b495b780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:25:17.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.956+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d109e32c10 con 0x55d109da0a30 2023-10-30T20:25:17.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.956+0000 7fa1b495b780 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:25:17.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.956+0000 7fa1b495b780 10 monclient: _renew_subs 2023-10-30T20:25:17.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.956+0000 7fa1b495b780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:25:17.535 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-10-30T20:25:17.536 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-10-30T20:25:17.536 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-10-30T20:25:17.536 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.536 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-10-30T20:25:17.536 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-10-30T20:25:17.536 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T20:25:17.537 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-10-30T20:25:17.537 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.537 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T20:25:17.537 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.537 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T20:25:17.537 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.538 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T20:25:17.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T20:25:17.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T20:25:17.538 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.539 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T20:25:17.539 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.539 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T20:25:17.539 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.539 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T20:25:17.539 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.540 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T20:25:17.540 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T20:25:17.540 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T20:25:17.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T20:25:17.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.541 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T20:25:17.541 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ list_parallel: /' 2023-10-30T20:25:17.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T20:25:17.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-10-30T20:25:17.541 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T20:25:17.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T20:25:17.542 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2023-10-30T20:25:17.542 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-10-30T20:25:17.542 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-10-30T20:25:17.543 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.543 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T20:25:17.543 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.543 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T20:25:17.543 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2023-10-30T20:25:17.544 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.544 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.544 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.544 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.544 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.545 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.545 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.545 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.545 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.545 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.545 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.546 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.546 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.546 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.546 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.546 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.547 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.547 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T20:25:17.547 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.547 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.547 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T20:25:17.548 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.548 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.548 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T20:25:17.548 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.548 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.548 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T20:25:17.549 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.549 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.549 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T20:25:17.549 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T20:25:17.549 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.550 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.550 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.550 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.550 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.550 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.550 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.551 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T20:25:17.551 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.551 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.551 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.551 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.551 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.552 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.552 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.552 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.552 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.552 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.553 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.553 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.553 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.553 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.553 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.554 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.554 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.554 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.554 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.554 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.555 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.555 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.555 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.555 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.555 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.555 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.556 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92114/exe 2023-10-30T20:25:17.556 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.556 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92214/exe 2023-10-30T20:25:17.556 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.556 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T20:25:17.557 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T20:25:17.557 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T20:25:17.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T20:25:17.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.558 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T20:25:17.558 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T20:25:17.558 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.558 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T20:25:17.558 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.559 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T20:25:17.559 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.559 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.559 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.559 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.560 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.560 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.560 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.560 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.560 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.561 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.561 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.561 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T20:25:17.561 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.561 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T20:25:17.561 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-10-30T20:25:17.562 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_open_pools_parallel 2023-10-30T20:25:17.562 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ open_pools_parallel: /' 2023-10-30T20:25:17.562 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.562 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.562 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.563 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.563 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.563 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.563 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.564 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.564 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.564 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-10-30T20:25:17.564 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_c_read_operations 2023-10-30T20:25:17.564 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2023-10-30T20:25:17.565 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_c_read_operations: /' 2023-10-30T20:25:17.565 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2023-10-30T20:25:17.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T20:25:17.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T20:25:17.566 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T20:25:17.566 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T20:25:17.566 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T20:25:17.566 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T20:25:17.566 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-10-30T20:25:17.566 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T20:25:17.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T20:25:17.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-10-30T20:25:17.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-10-30T20:25:17.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-10-30T20:25:17.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T20:25:17.568 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-10-30T20:25:17.568 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.568 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T20:25:17.568 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.568 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-10-30T20:25:17.568 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T20:25:17.569 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T20:25:17.569 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T20:25:17.569 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-10-30T20:25:17.569 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T20:25:17.569 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T20:25:17.569 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92248/exe 2023-10-30T20:25:17.570 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-10-30T20:25:17.570 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-10-30T20:25:17.570 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-10-30T20:25:17.570 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-10-30T20:25:17.570 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-10-30T20:25:17.570 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-10-30T20:25:17.571 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-10-30T20:25:17.571 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T20:25:17.571 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-10-30T20:25:17.571 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T20:25:17.571 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-10-30T20:25:17.572 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_delete_pools_parallel 2023-10-30T20:25:17.572 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2023-10-30T20:25:17.572 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ delete_pools_parallel: /' 2023-10-30T20:25:17.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.956+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d109ed2990 con 0x55d109da0a30 2023-10-30T20:25:17.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.956+0000 7fa1b495b780 1 librados: init done 2023-10-30T20:25:17.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:16.956+0000 7fa1b495b780 10 librados: wait_for_osdmap waiting 2023-10-30T20:25:17.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.104+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fa1a805d070 con 0x55d109da0a30 2023-10-30T20:25:17.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.104+0000 7fa1a37fe640 1 --2- 172.21.15.69:0/1144027014 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa19406aa00 0x7fa19406cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:25:17.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.105+0000 7fa1b245f640 1 --2- 172.21.15.69:0/1144027014 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa19406aa00 0x7fa19406cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:25:17.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.105+0000 7fa1b245f640 10 monclient: get_auth_request con 0x7fa19406aa00 auth_method 0 2023-10-30T20:25:17.574 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.105+0000 7fa1b245f640 1 --2- 172.21.15.69:0/1144027014 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa19406aa00 0x7fa19406cec0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4005770 tx=0x7fa1a40056e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:25:17.574 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.105+0000 7fa1a37fe640 10 client.4367.objecter ms_handle_connect 0x7fa19406aa00 2023-10-30T20:25:17.574 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.109+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fa1a8065040 con 0x55d109da0a30 2023-10-30T20:25:17.574 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.109+0000 7fa1a37fe640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_map(22..22 src has 1..22) v4 2023-10-30T20:25:17.574 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.109+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T20:25:17.575 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.109+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding full epoch 22 2023-10-30T20:25:17.575 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.109+0000 7fa1a37fe640 20 client.4367.objecter dump_active .. 0 homeless 2023-10-30T20:25:17.575 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.109+0000 7fa1b495b780 10 librados: wait_for_osdmap done waiting 2023-10-30T20:25:17.575 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.109+0000 7fa1b495b780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:25:17.575 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.109+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55d109d14e50 con 0x55d109da0a30 2023-10-30T20:25:17.576 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.122+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1a8064040 con 0x55d109da0a30 2023-10-30T20:25:17.576 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.122+0000 7fa1a37fe640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T20:25:17.576 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.122+0000 7fa1a3fff640 10 client.4367.objecter _get_latest_version latest 22, have it 2023-10-30T20:25:17.576 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.122+0000 7fa1b495b780 10 client.4367.objecter create_pool name=ceph_test_rados_api_asio 2023-10-30T20:25:17.576 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.122+0000 7fa1b495b780 10 client.4367.objecter pool_op_submit 1 2023-10-30T20:25:17.577 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.122+0000 7fa1b495b780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:25:18.200 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:17.122+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55d109d260d0 con 0x55d109da0a30 2023-10-30T20:25:18.200 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.185+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== mon.2 v2:172.21.15.69:3301/0 7 ==== pool_op_reply(tid 1 (0) Success v23) v1 ==== 43+0+0 (secure 0 0 0) 0x7fa1a805b070 con 0x55d109da0a30 2023-10-30T20:25:18.200 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.185+0000 7fa1a37fe640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 pool_op_reply(tid 1 (0) Success v23) v1 2023-10-30T20:25:18.201 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.185+0000 7fa1a37fe640 10 client.4367.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v23) v1 2023-10-30T20:25:18.201 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.185+0000 7fa1a37fe640 10 client.4367.objecter have request 1 at 0x55d109d65950 Op: create 2023-10-30T20:25:18.201 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.185+0000 7fa1a37fe640 20 client.4367.objecter waiting for client to reach epoch 23 before calling back 2023-10-30T20:25:18.201 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.185+0000 7fa1a37fe640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:25:18.201 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.185+0000 7fa1a37fe640 10 monclient: _renew_subs 2023-10-30T20:25:18.201 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.185+0000 7fa1a37fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:25:18.202 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.185+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=23}) v3 -- 0x7fa1a8094650 con 0x55d109da0a30 2023-10-30T20:25:18.202 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.185+0000 7fa1a37fe640 10 client.4367.objecter done 2023-10-30T20:25:18.202 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== mon.2 v2:172.21.15.69:3301/0 8 ==== osd_map(23..23 src has 1..23) v4 ==== 4713+0+0 (secure 0 0 0) 0x7fa1a8097b80 con 0x55d109da0a30 2023-10-30T20:25:18.202 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1a37fe640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_map(23..23 src has 1..23) v4 2023-10-30T20:25:18.202 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map got epochs [23,23] > 22 2023-10-30T20:25:18.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 23 2023-10-30T20:25:18.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1a37fe640 20 client.4367.objecter dump_active .. 0 homeless 2023-10-30T20:25:18.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 10 librados: set snap read head -> 1 2023-10-30T20:25:18.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 10 librados: writefull oid=exist nspace= 2023-10-30T20:25:18.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109e20e20 2023-10-30T20:25:18.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 23 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:18.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:18.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:18.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 1 --2- 172.21.15.69:0/1144027014 >> v2:172.21.15.69:6801/2068724062 conn(0x55d109e217d0 0x55d109e26110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:25:18.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:18.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid exist '@10' '@10' [writefull 0~5 in=5b] tid 2 osd.0 2023-10-30T20:25:18.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:18.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 2 2023-10-30T20:25:18.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 15 client.4367.objecter _send_op 2 to 10.1 on osd.0 2023-10-30T20:25:26.530 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:2 10.1 10:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) v8 -- 0x55d109e2fe80 con 0x55d109e217d0 2023-10-30T20:25:26.530 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:26.530 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.198+0000 7fa1b495b780 5 client.4367.objecter 1 in flight 2023-10-30T20:25:26.531 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.199+0000 7fa1b2c60640 1 --2- 172.21.15.69:0/1144027014 >> v2:172.21.15.69:6801/2068724062 conn(0x55d109e217d0 0x55d109e26110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:25:26.531 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.199+0000 7fa1b2c60640 10 monclient: get_auth_request con 0x55d109e217d0 auth_method 0 2023-10-30T20:25:26.531 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.202+0000 7fa1a37fe640 10 client.4367.objecter ms_handle_connect 0x55d109e217d0 2023-10-30T20:25:26.531 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:18.202+0000 7fa1b2c60640 1 --2- 172.21.15.69:0/1144027014 >> v2:172.21.15.69:6801/2068724062 conn(0x55d109e217d0 0x55d109e26110 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:25:26.531 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:19.829+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:26.532 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:19.829+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:26.532 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:19.829+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:26.532 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:19.829+0000 7fa1a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:25:26.532 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:20.829+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:26.532 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:20.829+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:26.532 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:20.829+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:26.533 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:21.829+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:26.533 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:21.829+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:26.533 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:21.829+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:26.533 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:21.957+0000 7fa1b46c3640 10 client.4367.objecter tick 2023-10-30T20:25:26.533 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:22.829+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:26.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:22.829+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:26.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:22.829+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:26.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:23.829+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:26.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:23.829+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:26.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:23.829+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:26.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:24.830+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:26.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:24.830+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:26.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:24.830+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:26.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:25.830+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:26.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:25.830+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:26.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:25.830+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:28.550 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.528+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 1 ==== osd_map(24..26 src has 1..26) v4 ==== 15422+0+0 (crc 0 0 0) 0x7fa1ac001970 con 0x55d109e217d0 2023-10-30T20:25:28.551 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.528+0000 7fa1a37fe640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_map(24..26 src has 1..26) v4 2023-10-30T20:25:28.551 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.528+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map got epochs [24,26] > 23 2023-10-30T20:25:28.551 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.528+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 24 2023-10-30T20:25:28.551 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.528+0000 7fa1a37fe640 10 client.4367.objecter checking op 2 2023-10-30T20:25:28.551 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.528+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 24 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:28.552 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.528+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:28.552 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.528+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 25 2023-10-30T20:25:28.552 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.529+0000 7fa1a37fe640 10 client.4367.objecter checking op 2 2023-10-30T20:25:28.552 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.529+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 25 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:28.552 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.529+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:28.553 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.529+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 26 2023-10-30T20:25:28.553 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.529+0000 7fa1a37fe640 10 client.4367.objecter checking op 2 2023-10-30T20:25:28.553 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.529+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 26 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:28.553 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.529+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:28.554 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.529+0000 7fa1a37fe640 20 client.4367.objecter dump_active .. 0 homeless 2023-10-30T20:25:28.554 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.529+0000 7fa1a37fe640 20 client.4367.objecter 2 10.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2023-10-30T20:25:28.554 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.830+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:28.554 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.830+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:28.554 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.830+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:28.554 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:26.958+0000 7fa1b46c3640 10 client.4367.objecter tick 2023-10-30T20:25:28.555 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:27.830+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:28.555 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:27.830+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:28.555 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:27.830+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:28.555 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.548+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 2 ==== osd_op_reply(2 exist [writefull 0~5] v26'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:28.555 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.548+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(2 exist [writefull 0~5] v26'1 uv1 ondisk = 0) v8 2023-10-30T20:25:28.556 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:28.556 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 2 ondisk uv 1 in 10.1 attempt 0 2023-10-30T20:25:28.556 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 0 2023-10-30T20:25:28.556 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 2 2023-10-30T20:25:28.556 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 2 2023-10-30T20:25:28.804 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:50.862 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:28.826 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:50.884 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:28.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:28.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 2 2023-10-30T20:25:28.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b2c60640 5 client.4367.objecter 0 in flight 2023-10-30T20:25:28.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2023-10-30T20:25:28.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 10 librados: Objecter returned from writefull r=0 2023-10-30T20:25:28.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109e20e20 2023-10-30T20:25:28.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 26 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:28.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:28.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:28.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:28.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid exist '@10' '@10' [read 0~256] tid 3 osd.0 2023-10-30T20:25:28.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:28.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 3 2023-10-30T20:25:28.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 15 client.4367.objecter _send_op 3 to 10.1 on osd.0 2023-10-30T20:25:28.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:3 10.1 10:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26) v8 -- 0x55d109d63530 con 0x55d109e217d0 2023-10-30T20:25:28.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:28.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 5 client.4367.objecter 1 in flight 2023-10-30T20:25:28.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109d638d0 2023-10-30T20:25:28.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 26 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:28.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 20 client.4367.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-10-30T20:25:28.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.a7bbaa1 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:28.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:28.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid noexist '@10' '@10' [read 0~256] tid 4 osd.0 2023-10-30T20:25:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 4 2023-10-30T20:25:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 15 client.4367.objecter _send_op 4 to 10.1 on osd.0 2023-10-30T20:25:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:4 10.1 10:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26) v8 -- 0x55d109d63d70 con 0x55d109e217d0 2023-10-30T20:25:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 5 2023-10-30T20:25:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.549+0000 7fa1b495b780 5 client.4367.objecter 2 in flight 2023-10-30T20:25:28.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.830+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:28.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.830+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:28.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.830+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:28.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:28.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T20:25:28.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:28.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 3 ondisk uv 1 in 10.1 attempt 0 2023-10-30T20:25:28.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 5 2023-10-30T20:25:28.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 3 ==== osd_map(27..28 src has 1..28) v4 ==== 2347+0+0 (crc 0 0 0) 0x7fa1ac014da0 con 0x55d109e217d0 2023-10-30T20:25:28.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 3 2023-10-30T20:25:28.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 3 2023-10-30T20:25:28.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1a37fe640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_map(27..28 src has 1..28) v4 2023-10-30T20:25:28.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 5 2023-10-30T20:25:28.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 3 2023-10-30T20:25:28.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 5 client.4367.objecter 1 in flight 2023-10-30T20:25:28.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map got epochs [27,28] > 26 2023-10-30T20:25:28.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 27 2023-10-30T20:25:28.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 5 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:28.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T20:25:28.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:28.875 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1a37fe640 10 client.4367.objecter checking op 4 2023-10-30T20:25:28.875 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 27 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:28.875 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-10-30T20:25:28.875 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 28 2023-10-30T20:25:28.875 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1a37fe640 10 client.4367.objecter checking op 4 2023-10-30T20:25:28.875 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 28 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:28.876 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-10-30T20:25:28.876 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1a37fe640 20 client.4367.objecter dump_active .. 0 homeless 2023-10-30T20:25:28.876 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1a37fe640 20 client.4367.objecter 4 10.a7bbaa1 osd.0 noexist [read 0~256] 2023-10-30T20:25:28.876 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (320 ms) 2023-10-30T20:25:28.876 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2023-10-30T20:25:28.876 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 4 ondisk uv 0 in 10.1 attempt 0 2023-10-30T20:25:28.877 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 10 client.4367.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T20:25:28.877 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 0 2023-10-30T20:25:28.877 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 4 2023-10-30T20:25:28.877 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 4 2023-10-30T20:25:28.877 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:28.877 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 4 2023-10-30T20:25:28.878 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b2c60640 5 client.4367.objecter 0 in flight 2023-10-30T20:25:28.878 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109d638d0 2023-10-30T20:25:28.878 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 28 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:28.878 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:28.878 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:28.879 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:28.879 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid exist '@10' '@10' [read 0~256] tid 5 osd.0 2023-10-30T20:25:28.879 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:28.879 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 5 2023-10-30T20:25:28.879 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 15 client.4367.objecter _send_op 5 to 10.1 on osd.0 2023-10-30T20:25:28.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:5 10.1 10:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28) v8 -- 0x55d109d64160 con 0x55d109e217d0 2023-10-30T20:25:28.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:28.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 5 client.4367.objecter 1 in flight 2023-10-30T20:25:28.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109e20e20 2023-10-30T20:25:28.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 28 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:28.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 20 client.4367.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-10-30T20:25:28.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.a7bbaa1 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:28.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:28.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid noexist '@10' '@10' [read 0~256] tid 6 osd.0 2023-10-30T20:25:28.881 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:50.906 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:28.882 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:50.938 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:28.884 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:50.943 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:28.885 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:50.912 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:28.951 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:28.951 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 6 2023-10-30T20:25:28.951 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 15 client.4367.objecter _send_op 6 to 10.1 on osd.0 2023-10-30T20:25:28.951 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:6 10.1 10:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28) v8 -- 0x55d109e26650 con 0x55d109e217d0 2023-10-30T20:25:28.952 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 5 2023-10-30T20:25:28.952 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.869+0000 7fa1b495b780 5 client.4367.objecter 2 in flight 2023-10-30T20:25:28.952 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 6 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:28.952 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T20:25:28.952 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:28.953 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 5 ondisk uv 1 in 10.1 attempt 0 2023-10-30T20:25:28.953 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 5 2023-10-30T20:25:28.953 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 5 2023-10-30T20:25:28.953 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 5 2023-10-30T20:25:28.953 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 5 2023-10-30T20:25:28.953 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 5 2023-10-30T20:25:28.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 5 client.4367.objecter 1 in flight 2023-10-30T20:25:28.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 7 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:28.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T20:25:28.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:28.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 6 ondisk uv 0 in 10.1 attempt 0 2023-10-30T20:25:28.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 10 client.4367.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T20:25:28.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 0 2023-10-30T20:25:28.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 6 2023-10-30T20:25:28.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 6 2023-10-30T20:25:28.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:28.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 6 2023-10-30T20:25:28.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b2c60640 5 client.4367.objecter 0 in flight 2023-10-30T20:25:28.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (80 ms) 2023-10-30T20:25:28.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2023-10-30T20:25:28.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109e20e20 2023-10-30T20:25:28.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 28 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:28.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b495b780 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:28.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:28.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:28.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid exist '@10' '@10' [read 0~256] tid 7 osd.0 2023-10-30T20:25:28.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:28.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 7 2023-10-30T20:25:28.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b495b780 15 client.4367.objecter _send_op 7 to 10.1 on osd.0 2023-10-30T20:25:28.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.949+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:7 10.1 10:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28) v8 -- 0x55d109d638d0 con 0x55d109e217d0 2023-10-30T20:25:28.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.950+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:28.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.950+0000 7fa1b495b780 5 client.4367.objecter 1 in flight 2023-10-30T20:25:28.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.950+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109e270b0 2023-10-30T20:25:28.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.950+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 28 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:28.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.950+0000 7fa1b495b780 20 client.4367.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-10-30T20:25:28.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.950+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.a7bbaa1 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:28.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.950+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:28.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.950+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid noexist '@10' '@10' [read 0~256] tid 8 osd.0 2023-10-30T20:25:28.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.950+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:28.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.950+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 8 2023-10-30T20:25:28.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.950+0000 7fa1b495b780 15 client.4367.objecter _send_op 8 to 10.1 on osd.0 2023-10-30T20:25:28.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.950+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:8 10.1 10:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28) v8 -- 0x55d109e27550 con 0x55d109e217d0 2023-10-30T20:25:28.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.950+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 5 2023-10-30T20:25:28.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.950+0000 7fa1b495b780 5 client.4367.objecter 2 in flight 2023-10-30T20:25:28.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 8 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:28.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T20:25:28.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:28.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 7 ondisk uv 1 in 10.1 attempt 0 2023-10-30T20:25:28.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 5 2023-10-30T20:25:28.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 7 2023-10-30T20:25:28.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 7 2023-10-30T20:25:28.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 5 2023-10-30T20:25:28.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 7 2023-10-30T20:25:28.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 5 client.4367.objecter 1 in flight 2023-10-30T20:25:28.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (23 ms) 2023-10-30T20:25:28.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2023-10-30T20:25:28.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 9 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:28.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T20:25:28.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:28.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 8 ondisk uv 0 in 10.1 attempt 0 2023-10-30T20:25:28.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 10 client.4367.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T20:25:28.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 0 2023-10-30T20:25:28.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 8 2023-10-30T20:25:28.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 8 2023-10-30T20:25:28.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:28.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 8 2023-10-30T20:25:28.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b2c60640 5 client.4367.objecter 0 in flight 2023-10-30T20:25:28.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T20:25:28.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 20 librados: queue_aio_write 0x55d109cb6db0 completion 0x55d109e31bc0 write_seq 1 2023-10-30T20:25:28.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109e20e20 2023-10-30T20:25:28.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 28 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:28.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:28.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:28.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:28.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid exist '@10' '@10' [write 0~5 in=5b] tid 9 osd.0 2023-10-30T20:25:28.984 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:28.984 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 9 2023-10-30T20:25:28.991 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:51.049 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:28.999 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:51.026 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:29.010 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:51.069 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:31.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 15 client.4367.objecter _send_op 9 to 10.1 on osd.0 2023-10-30T20:25:31.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:9 10.1 10:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) v8 -- 0x55d109e27940 con 0x55d109e217d0 2023-10-30T20:25:31.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:31.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 5 client.4367.objecter 1 in flight 2023-10-30T20:25:31.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:28.972+0000 7fa1b495b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T20:25:31.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:29.830+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:31.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:29.830+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:31.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:29.830+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:31.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:29.830+0000 7fa1a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:25:31.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:30.830+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:31.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:30.830+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:31.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:30.830+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:31.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 10 ==== osd_op_reply(9 exist [write 0~5] v28'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:31.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(9 exist [write 0~5] v28'2 uv2 ondisk = 0) v8 2023-10-30T20:25:31.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:31.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 9 ondisk uv 2 in 10.1 attempt 0 2023-10-30T20:25:31.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 0 2023-10-30T20:25:31.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 9 2023-10-30T20:25:31.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 9 2023-10-30T20:25:31.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:31.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 9 2023-10-30T20:25:31.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b2c60640 5 client.4367.objecter 0 in flight 2023-10-30T20:25:31.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b2c60640 20 librados: complete_aio_write 0x55d109e31bc0 2023-10-30T20:25:31.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (2251 ms) 2023-10-30T20:25:31.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2023-10-30T20:25:31.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T20:25:31.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 20 librados: queue_aio_write 0x55d109cb6db0 completion 0x55d109e27cb0 write_seq 2 2023-10-30T20:25:31.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109e20e20 2023-10-30T20:25:31.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 28 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:31.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:31.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:33.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:33.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid exist '@10' '@10' [write 0~5 in=5b] tid 10 osd.0 2023-10-30T20:25:33.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:33.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 10 2023-10-30T20:25:33.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 15 client.4367.objecter _send_op 10 to 10.1 on osd.0 2023-10-30T20:25:33.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:10 10.1 10:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) v8 -- 0x55d109e27940 con 0x55d109e217d0 2023-10-30T20:25:33.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:33.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 5 client.4367.objecter 1 in flight 2023-10-30T20:25:33.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.223+0000 7fa1b495b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T20:25:33.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.376+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 11 ==== osd_map(29..30 src has 1..30) v4 ==== 1631+0+0 (crc 0 0 0) 0x7fa1ac014ad0 con 0x55d109e217d0 2023-10-30T20:25:33.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.377+0000 7fa1a37fe640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_map(29..30 src has 1..30) v4 2023-10-30T20:25:33.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.377+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map got epochs [29,30] > 28 2023-10-30T20:25:33.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.377+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 29 2023-10-30T20:25:33.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.377+0000 7fa1a37fe640 10 client.4367.objecter checking op 10 2023-10-30T20:25:33.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.377+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 29 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:33.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.377+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:33.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.377+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 30 2023-10-30T20:25:33.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.377+0000 7fa1a37fe640 10 client.4367.objecter checking op 10 2023-10-30T20:25:33.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.377+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 30 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:33.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.377+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:33.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.377+0000 7fa1a37fe640 20 client.4367.objecter dump_active .. 0 homeless 2023-10-30T20:25:33.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.377+0000 7fa1a37fe640 20 client.4367.objecter 10 10.d3a59201 osd.0 exist [write 0~5 in=5b] 2023-10-30T20:25:33.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.830+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:33.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.830+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:33.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.830+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:33.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:31.959+0000 7fa1b46c3640 10 client.4367.objecter tick 2023-10-30T20:25:33.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:32.830+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:33.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:32.830+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:33.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:32.830+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:33.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:33.830+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:33.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:33.830+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:38.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:33.830+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:38.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:34.830+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:38.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:34.830+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:38.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:34.830+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:38.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:35.830+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:38.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:35.830+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:38.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:35.830+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:38.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:36.830+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:38.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:36.830+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:38.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:36.830+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:38.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:36.959+0000 7fa1b46c3640 10 client.4367.objecter tick 2023-10-30T20:25:38.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:37.830+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:38.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:37.830+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:38.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:37.830+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:38.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 12 ==== osd_op_reply(10 exist [write 0~5] v30'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:38.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(10 exist [write 0~5] v30'3 uv3 ondisk = 0) v8 2023-10-30T20:25:38.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:38.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 10 ondisk uv 3 in 10.1 attempt 0 2023-10-30T20:25:38.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 0 2023-10-30T20:25:38.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 10 2023-10-30T20:25:38.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 10 2023-10-30T20:25:38.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:38.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 10 2023-10-30T20:25:38.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b2c60640 5 client.4367.objecter 0 in flight 2023-10-30T20:25:38.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b2c60640 20 librados: complete_aio_write 0x55d109e27cb0 2023-10-30T20:25:38.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (7534 ms) 2023-10-30T20:25:38.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2023-10-30T20:25:38.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T20:25:38.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 20 librados: queue_aio_write 0x55d109cb6db0 completion 0x55d109e32050 write_seq 3 2023-10-30T20:25:38.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109e20e20 2023-10-30T20:25:38.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 30 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:38.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:38.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:41.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:41.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid exist '@10' '@10' [write 0~5 in=5b] tid 11 osd.0 2023-10-30T20:25:41.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:41.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 11 2023-10-30T20:25:41.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 15 client.4367.objecter _send_op 11 to 10.1 on osd.0 2023-10-30T20:25:41.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:11 10.1 10:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d109f0a1f0 con 0x55d109e217d0 2023-10-30T20:25:41.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:41.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 5 client.4367.objecter 1 in flight 2023-10-30T20:25:41.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.757+0000 7fa1b495b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T20:25:41.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.831+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:41.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.831+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:41.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:38.831+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:41.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:39.831+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:41.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:39.831+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:41.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:39.831+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:41.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:39.831+0000 7fa1a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:25:41.193 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:40.831+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:41.193 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:40.831+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:41.193 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:40.831+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:41.193 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 13 ==== osd_map(31..39 src has 1..39) v4 ==== 5704+0+0 (crc 0 0 0) 0x7fa1ac001dc0 con 0x55d109e217d0 2023-10-30T20:25:41.194 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_map(31..39 src has 1..39) v4 2023-10-30T20:25:41.194 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map got epochs [31,39] > 30 2023-10-30T20:25:41.194 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 31 2023-10-30T20:25:41.194 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 10 client.4367.objecter checking op 11 2023-10-30T20:25:41.194 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 31 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:41.195 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:41.195 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 32 2023-10-30T20:25:41.195 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 10 client.4367.objecter checking op 11 2023-10-30T20:25:41.195 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 32 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:41.195 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:41.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 33 2023-10-30T20:25:41.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 10 client.4367.objecter checking op 11 2023-10-30T20:25:41.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 33 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:41.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:41.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 34 2023-10-30T20:25:41.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 10 client.4367.objecter checking op 11 2023-10-30T20:25:41.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 34 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:41.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:41.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 35 2023-10-30T20:25:41.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 10 client.4367.objecter checking op 11 2023-10-30T20:25:41.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 35 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:41.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:41.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 36 2023-10-30T20:25:41.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 10 client.4367.objecter checking op 11 2023-10-30T20:25:41.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 36 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:41.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:41.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 37 2023-10-30T20:25:41.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 10 client.4367.objecter checking op 11 2023-10-30T20:25:41.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 37 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:41.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:41.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 38 2023-10-30T20:25:41.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 10 client.4367.objecter checking op 11 2023-10-30T20:25:41.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 38 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:41.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:41.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 39 2023-10-30T20:25:41.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 10 client.4367.objecter checking op 11 2023-10-30T20:25:41.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 39 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:41.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:41.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter dump_active .. 0 homeless 2023-10-30T20:25:41.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.188+0000 7fa1a37fe640 20 client.4367.objecter 11 10.d3a59201 osd.0 exist [write 0~5 in=5b] 2023-10-30T20:25:43.331 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.831+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:43.331 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.831+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:43.331 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.831+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:43.331 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:41.960+0000 7fa1b46c3640 10 client.4367.objecter tick 2023-10-30T20:25:43.331 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:42.831+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:43.332 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:42.831+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:43.332 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:42.831+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:43.332 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 14 ==== osd_op_reply(11 exist [write 0~5] v39'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:43.332 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(11 exist [write 0~5] v39'4 uv4 ondisk = 0) v8 2023-10-30T20:25:43.332 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:43.332 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 11 ondisk uv 4 in 10.1 attempt 0 2023-10-30T20:25:43.333 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 0 2023-10-30T20:25:43.333 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 11 2023-10-30T20:25:43.333 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 11 2023-10-30T20:25:43.333 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:43.333 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 11 2023-10-30T20:25:43.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b2c60640 5 client.4367.objecter 0 in flight 2023-10-30T20:25:43.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b2c60640 20 librados: complete_aio_write 0x55d109e32050 2023-10-30T20:25:43.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (4572 ms) 2023-10-30T20:25:43.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2023-10-30T20:25:43.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109e20e20 2023-10-30T20:25:43.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 39 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:43.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:43.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:43.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:43.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid exist '@10' '@10' [read 0~0] tid 12 osd.0 2023-10-30T20:25:43.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:43.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 12 2023-10-30T20:25:43.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 15 client.4367.objecter _send_op 12 to 10.1 on osd.0 2023-10-30T20:25:43.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:12 10.1 10:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v8 -- 0x55d109ee8020 con 0x55d109e217d0 2023-10-30T20:25:43.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:43.859 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 5 client.4367.objecter 1 in flight 2023-10-30T20:25:43.859 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109ee84a0 2023-10-30T20:25:43.859 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 39 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:43.860 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 20 client.4367.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-10-30T20:25:43.860 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.a7bbaa1 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:43.860 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:43.860 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid noexist '@10' '@10' [read 0~0] tid 13 osd.0 2023-10-30T20:25:43.860 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:43.861 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 13 2023-10-30T20:25:43.861 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 15 client.4367.objecter _send_op 13 to 10.1 on osd.0 2023-10-30T20:25:43.861 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:13 10.1 10:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v8 -- 0x55d109ee8940 con 0x55d109e217d0 2023-10-30T20:25:43.861 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 5 2023-10-30T20:25:43.861 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.329+0000 7fa1b495b780 5 client.4367.objecter 2 in flight 2023-10-30T20:25:43.861 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.831+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:43.862 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.831+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:43.862 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.831+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:43.862 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 16 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:43.862 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T20:25:43.862 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:43.863 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 12 ondisk uv 4 in 10.1 attempt 0 2023-10-30T20:25:43.863 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 5 2023-10-30T20:25:43.863 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 12 2023-10-30T20:25:43.863 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 12 2023-10-30T20:25:43.863 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 5 2023-10-30T20:25:43.863 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 12 2023-10-30T20:25:43.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1b2c60640 5 client.4367.objecter 1 in flight 2023-10-30T20:25:43.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 15 ==== osd_map(40..42 src has 1..43) v4 ==== 2294+0+0 (crc 0 0 0) 0x7fa1ac004a20 con 0x55d109e217d0 2023-10-30T20:25:43.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1a37fe640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_map(40..42 src has 1..43) v4 2023-10-30T20:25:43.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map got epochs [40,42] > 39 2023-10-30T20:25:43.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 40 2023-10-30T20:25:43.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 17 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:43.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T20:25:43.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.857+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:43.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1a37fe640 10 client.4367.objecter checking op 13 2023-10-30T20:25:43.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 40 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:43.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-10-30T20:25:43.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 41 2023-10-30T20:25:43.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1a37fe640 10 client.4367.objecter checking op 13 2023-10-30T20:25:43.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 41 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:43.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-10-30T20:25:43.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 42 2023-10-30T20:25:43.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1a37fe640 10 client.4367.objecter checking op 13 2023-10-30T20:25:43.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 42 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:43.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-10-30T20:25:43.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1a37fe640 20 client.4367.objecter dump_active .. 0 homeless 2023-10-30T20:25:43.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1a37fe640 20 client.4367.objecter 13 10.a7bbaa1 osd.0 noexist [read 0~0] 2023-10-30T20:25:43.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 13 ondisk uv 0 in 10.1 attempt 0 2023-10-30T20:25:43.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b2c60640 10 client.4367.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T20:25:43.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 0 2023-10-30T20:25:43.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 13 2023-10-30T20:25:43.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 13 2023-10-30T20:25:43.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:43.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 13 2023-10-30T20:25:43.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b2c60640 5 client.4367.objecter 0 in flight 2023-10-30T20:25:43.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (529 ms) 2023-10-30T20:25:43.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2023-10-30T20:25:43.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109e20e20 2023-10-30T20:25:43.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 42 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:44.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:44.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:44.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:44.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid exist '@10' '@10' [read 0~0] tid 14 osd.0 2023-10-30T20:25:44.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:44.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 14 2023-10-30T20:25:44.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 15 client.4367.objecter _send_op 14 to 10.1 on osd.0 2023-10-30T20:25:44.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:14 10.1 10:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55d109ee8020 con 0x55d109e217d0 2023-10-30T20:25:44.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:44.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 5 client.4367.objecter 1 in flight 2023-10-30T20:25:44.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109ee85b0 2023-10-30T20:25:44.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 42 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:44.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 20 client.4367.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-10-30T20:25:44.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.a7bbaa1 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:44.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:44.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid noexist '@10' '@10' [read 0~0] tid 15 osd.0 2023-10-30T20:25:44.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:44.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 15 2023-10-30T20:25:44.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 15 client.4367.objecter _send_op 15 to 10.1 on osd.0 2023-10-30T20:25:44.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:15 10.1 10:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55d109ee8a50 con 0x55d109e217d0 2023-10-30T20:25:44.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 5 2023-10-30T20:25:44.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:43.858+0000 7fa1b495b780 5 client.4367.objecter 2 in flight 2023-10-30T20:25:44.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 19 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:44.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T20:25:44.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:44.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 14 ondisk uv 4 in 10.1 attempt 0 2023-10-30T20:25:44.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 5 2023-10-30T20:25:44.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 14 2023-10-30T20:25:44.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 14 2023-10-30T20:25:44.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 5 2023-10-30T20:25:44.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 14 2023-10-30T20:25:44.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 5 client.4367.objecter 1 in flight 2023-10-30T20:25:44.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 20 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:44.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T20:25:44.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:44.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 15 ondisk uv 0 in 10.1 attempt 0 2023-10-30T20:25:44.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 10 client.4367.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T20:25:44.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 0 2023-10-30T20:25:44.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 15 2023-10-30T20:25:44.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 15 2023-10-30T20:25:44.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:44.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 15 2023-10-30T20:25:44.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b2c60640 5 client.4367.objecter 0 in flight 2023-10-30T20:25:44.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (310 ms) 2023-10-30T20:25:44.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2023-10-30T20:25:44.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 18 ==== osd_map(43..43 src has 1..43) v4 ==== 691+0+0 (crc 0 0 0) 0x7fa1ac014730 con 0x55d109e217d0 2023-10-30T20:25:44.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1a37fe640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_map(43..43 src has 1..43) v4 2023-10-30T20:25:44.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map got epochs [43,43] > 42 2023-10-30T20:25:44.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 43 2023-10-30T20:25:44.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1a37fe640 20 client.4367.objecter dump_active .. 0 homeless 2023-10-30T20:25:44.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109e20e20 2023-10-30T20:25:44.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 43 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:44.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:44.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:44.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:44.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid exist '@10' '@10' [read 0~0] tid 16 osd.0 2023-10-30T20:25:44.568 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:44.569 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 16 2023-10-30T20:25:44.569 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 15 client.4367.objecter _send_op 16 to 10.1 on osd.0 2023-10-30T20:25:44.569 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:16 10.1 10:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) v8 -- 0x55d109ee8c20 con 0x55d109e217d0 2023-10-30T20:25:44.569 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:44.570 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 5 client.4367.objecter 1 in flight 2023-10-30T20:25:44.570 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109f29b90 2023-10-30T20:25:44.570 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 43 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T20:25:44.570 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 20 client.4367.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-10-30T20:25:44.570 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.a7bbaa1 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:44.570 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:44.571 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid noexist '@10' '@10' [read 0~0] tid 17 osd.0 2023-10-30T20:25:44.571 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:44.571 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 17 2023-10-30T20:25:44.571 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 15 client.4367.objecter _send_op 17 to 10.1 on osd.0 2023-10-30T20:25:44.571 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:17 10.1 10:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) v8 -- 0x55d109f2a030 con 0x55d109e217d0 2023-10-30T20:25:44.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 5 2023-10-30T20:25:44.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.168+0000 7fa1b495b780 5 client.4367.objecter 2 in flight 2023-10-30T20:25:44.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 21 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:44.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T20:25:44.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:44.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 16 ondisk uv 4 in 10.1 attempt 0 2023-10-30T20:25:44.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 5 2023-10-30T20:25:44.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 16 2023-10-30T20:25:44.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 16 2023-10-30T20:25:44.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 5 2023-10-30T20:25:44.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 16 2023-10-30T20:25:44.574 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 5 client.4367.objecter 1 in flight 2023-10-30T20:25:44.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 22 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:44.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T20:25:44.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:44.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 17 ondisk uv 0 in 10.1 attempt 0 2023-10-30T20:25:44.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 10 client.4367.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T20:25:44.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 0 2023-10-30T20:25:44.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 17 2023-10-30T20:25:44.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 17 2023-10-30T20:25:44.711 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:44.711 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 17 2023-10-30T20:25:44.711 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (399 ms) 2023-10-30T20:25:44.711 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b2c60640 5 client.4367.objecter 0 in flight 2023-10-30T20:25:44.711 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2023-10-30T20:25:44.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b495b780 20 librados: queue_aio_write 0x55d109cb6db0 completion 0x55d109ee8020 write_seq 4 2023-10-30T20:25:44.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109e20e20 2023-10-30T20:25:44.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 43 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:44.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b495b780 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:44.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:44.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:44.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid exist '@10' '@10' [writefull 0~5 in=5b] tid 18 osd.0 2023-10-30T20:25:44.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:44.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 18 2023-10-30T20:25:44.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b495b780 15 client.4367.objecter _send_op 18 to 10.1 on osd.0 2023-10-30T20:25:44.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:18 10.1 10:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) v8 -- 0x55d109ee85b0 con 0x55d109e217d0 2023-10-30T20:25:44.714 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:44.714 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.567+0000 7fa1b495b780 5 client.4367.objecter 1 in flight 2023-10-30T20:25:44.714 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.707+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 23 ==== osd_map(44..44 src has 1..44) v4 ==== 715+0+0 (crc 0 0 0) 0x7fa1ac020440 con 0x55d109e217d0 2023-10-30T20:25:45.315 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.708+0000 7fa1a37fe640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_map(44..44 src has 1..44) v4 2023-10-30T20:25:45.315 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.708+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map got epochs [44,44] > 43 2023-10-30T20:25:45.315 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.708+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 44 2023-10-30T20:25:45.315 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.708+0000 7fa1a37fe640 10 client.4367.objecter checking op 18 2023-10-30T20:25:45.316 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.708+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 44 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:45.316 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.708+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:45.316 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.708+0000 7fa1a37fe640 20 client.4367.objecter dump_active .. 0 homeless 2023-10-30T20:25:45.316 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.708+0000 7fa1a37fe640 20 client.4367.objecter 18 10.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2023-10-30T20:25:45.316 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.831+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:45.316 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.831+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:45.317 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:44.831+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:45.317 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 24 ==== osd_op_reply(18 exist [writefull 0~5] v44'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:45.317 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(18 exist [writefull 0~5] v44'5 uv5 ondisk = 0) v8 2023-10-30T20:25:45.317 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:45.317 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 18 ondisk uv 5 in 10.1 attempt 0 2023-10-30T20:25:45.317 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 0 2023-10-30T20:25:45.318 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 18 2023-10-30T20:25:45.318 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 18 2023-10-30T20:25:45.318 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:45.318 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 18 2023-10-30T20:25:45.318 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b2c60640 5 client.4367.objecter 0 in flight 2023-10-30T20:25:45.318 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b2c60640 20 librados: complete_aio_write 0x55d109ee8020 2023-10-30T20:25:45.319 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (746 ms) 2023-10-30T20:25:45.319 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2023-10-30T20:25:45.319 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b495b780 20 librados: queue_aio_write 0x55d109cb6db0 completion 0x55d109ee8920 write_seq 5 2023-10-30T20:25:45.319 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109e20e20 2023-10-30T20:25:45.319 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 44 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:45.319 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b495b780 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:45.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:45.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:45.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid exist '@10' '@10' [writefull 0~5 in=5b] tid 19 osd.0 2023-10-30T20:25:45.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b495b780 20 client.4367.obj api_service: [==========] Running 4 tests from 1 test suite. 2023-10-30T20:25:45.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] Global test environment set-up. 2023-10-30T20:25:45.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] 4 tests from LibRadosService 2023-10-30T20:25:45.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2023-10-30T20:25:45.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5384 ms) 2023-10-30T20:25:45.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2023-10-30T20:25:45.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.RegisterLate (262 ms) 2023-10-30T20:25:45.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2023-10-30T20:25:45.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: cluster: 2023-10-30T20:25:45.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: id: 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:25:45.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: health: HEALTH_WARN 2023-10-30T20:25:45.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 13 pool(s) do not have an application enabled 2023-10-30T20:25:45.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-10-30T20:25:45.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: services: 2023-10-30T20:25:45.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-10-30T20:25:45.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mgr: y(active, since 2m), standbys: x 2023-10-30T20:25:45.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: osd: 8 osds: 8 up (since 88s), 8 in (since 2m) 2023-10-30T20:25:45.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: laundry: 2 daemons active (1 hosts) 2023-10-30T20:25:45.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-10-30T20:25:45.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: data: 2023-10-30T20:25:45.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pools: 26 pools, 777 pgs 2023-10-30T20:25:45.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: objects: 2 objects, 449 KiB 2023-10-30T20:25:45.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: usage: 213 MiB used, 720 GiB / 720 GiB avail 2023-10-30T20:25:45.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pgs: 87.001% pgs unknown 2023-10-30T20:25:45.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 11.840% pgs not active 2023-10-30T20:25:45.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 676 unknown 2023-10-30T20:25:45.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 92 creating+peering 2023-10-30T20:25:45.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 9 active+clean 2023-10-30T20:25:45.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-10-30T20:25:45.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-10-30T20:25:45.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: cluster: 2023-10-30T20:25:45.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: id: 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:25:45.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: health: HEALTH_WARN 2023-10-30T20:25:45.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 8 pool(s) do not have an application enabled 2023-10-30T20:25:45.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-10-30T20:25:45.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: services: 2023-10-30T20:25:45.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-10-30T20:25:45.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mgr: y(active, since 2m), standbys: x 2023-10-30T20:25:45.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: osd: 8 osds: 8 up (since 91s), 8 in (since 2m) 2023-10-30T20:25:45.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2023-10-30T20:25:45.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-10-30T20:25:45.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-10-30T20:25:45.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: data: 2023-10-30T20:25:45.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pools: 29 pools, 849 pgs 2023-10-30T20:25:45.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: objects: 2 objects, 449 KiB 2023-10-30T20:25:45.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: usage: 213 MiB used, 720 GiB / 720 GiB avail 2023-10-30T20:25:45.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pgs: 30.506% pgs unknown 2023-10-30T20:25:45.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 64.429% pgs not active 2023-10-30T20:25:45.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 470 creating+peering 2023-10-30T20:25:45.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 259 unknown 2023-10-30T20:25:45.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 77 creating+activating 2023-10-30T20:25:45.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 43 active+clean 2023-10-30T20:25:45.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-10-30T20:25:45.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-10-30T20:25:45.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.StatusFormat (3049 ms) 2023-10-30T20:25:45.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.Status 2023-10-30T20:25:45.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.Status (20096 ms) 2023-10-30T20:25:45.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] 4 tests from LibRadosService (28791 ms total) 2023-10-30T20:25:45.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-10-30T20:25:45.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] Global test environment tear-down 2023-10-30T20:25:45.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [==========] 4 tests from 1 test suite ran. (28791 ms total) 2023-10-30T20:25:45.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ PASSED ] 4 tests. 2023-10-30T20:25:45.986 INFO:tasks.workunit.client.0.smithi069.stdout:ecter get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:45.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 19 2023-10-30T20:25:45.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b495b780 15 client.4367.objecter _send_op 19 to 10.1 on osd.0 2023-10-30T20:25:45.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:19 10.1 10:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) v8 -- 0x55d109ee85b0 con 0x55d109e217d0 2023-10-30T20:25:45.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:45.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.313+0000 7fa1b495b780 5 client.4367.objecter 1 in flight 2023-10-30T20:25:45.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.831+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:45.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.831+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:45.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.831+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:45.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 25 ==== osd_op_reply(19 exist [writefull 0~5] v44'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:45.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(19 exist [writefull 0~5] v44'6 uv6 ondisk = 0) v8 2023-10-30T20:25:45.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:45.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 19 ondisk uv 6 in 10.1 attempt 0 2023-10-30T20:25:45.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 0 2023-10-30T20:25:45.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 19 2023-10-30T20:25:45.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 19 2023-10-30T20:25:45.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:45.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 19 2023-10-30T20:25:45.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b2c60640 5 client.4367.objecter 0 in flight 2023-10-30T20:25:45.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b2c60640 20 librados: complete_aio_write 0x55d109ee8920 2023-10-30T20:25:45.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (671 ms) 2023-10-30T20:25:45.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2023-10-30T20:25:45.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b495b780 20 librados: queue_aio_write 0x55d109cb6db0 completion 0x55d109ee8a30 write_seq 6 2023-10-30T20:25:45.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b495b780 10 client.4367.objecter _op_submit op 0x55d109e20e20 2023-10-30T20:25:45.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b495b780 20 client.4367.objecter _calc_target epoch 44 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:45.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b495b780 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:45.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b495b780 10 client.4367.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [0,2] primary 0 2023-10-30T20:25:45.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b495b780 20 client.4367.objecter _get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:45.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b495b780 10 client.4367.objecter _op_submit oid exist '@10' '@10' [writefull 0~5 in=5b] tid 20 osd.0 2023-10-30T20:25:45.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b495b780 20 client.4367.objecter get_session s=0x55d109e32a00 osd=0 3 2023-10-30T20:25:47.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b495b780 15 client.4367.objecter _session_op_assign 0 20 2023-10-30T20:25:47.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b495b780 15 client.4367.objecter _send_op 20 to 10.1 on osd.0 2023-10-30T20:25:47.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 --> v2:172.21.15.69:6801/2068724062 -- osd_op(unknown.0.0:20 10.1 10:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) v8 -- 0x55d109ee8020 con 0x55d109e217d0 2023-10-30T20:25:47.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:47.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:45.984+0000 7fa1b495b780 5 client.4367.objecter 1 in flight 2023-10-30T20:25:47.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:46.482+0000 7fa1a37fe640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 26 ==== osd_map(45..45 src has 1..45) v4 ==== 1043+0+0 (crc 0 0 0) 0x7fa1ac0208c0 con 0x55d109e217d0 2023-10-30T20:25:47.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:46.482+0000 7fa1a37fe640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_map(45..45 src has 1..45) v4 2023-10-30T20:25:47.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:46.482+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map got epochs [45,45] > 44 2023-10-30T20:25:47.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:46.482+0000 7fa1a37fe640 3 client.4367.objecter handle_osd_map decoding incremental epoch 45 2023-10-30T20:25:47.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:46.482+0000 7fa1a37fe640 10 client.4367.objecter checking op 20 2023-10-30T20:25:47.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:46.482+0000 7fa1a37fe640 20 client.4367.objecter _calc_target epoch 45 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T20:25:47.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:46.482+0000 7fa1a37fe640 20 client.4367.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-10-30T20:25:47.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:46.482+0000 7fa1a37fe640 20 client.4367.objecter dump_active .. 0 homeless 2023-10-30T20:25:47.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:46.482+0000 7fa1a37fe640 20 client.4367.objecter 20 10.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2023-10-30T20:25:47.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:46.831+0000 7fa1a27fc640 10 monclient: tick 2023-10-30T20:25:47.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:46.832+0000 7fa1a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:25:47.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:46.832+0000 7fa1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:25:47.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:46.961+0000 7fa1b46c3640 10 client.4367.objecter tick 2023-10-30T20:25:47.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 <== osd.0 v2:172.21.15.69:6801/2068724062 27 ==== osd_op_reply(20 exist [writefull 0~5] v45'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa1ac001b40 con 0x55d109e217d0 2023-10-30T20:25:47.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b2c60640 10 client.4367.objecter ms_dispatch 0x55d109cedd50 osd_op_reply(20 exist [writefull 0~5] v45'7 uv7 ondisk = 0) v8 2023-10-30T20:25:47.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b2c60640 10 client.4367.objecter in handle_osd_op_reply 2023-10-30T20:25:47.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b2c60640 7 client.4367.objecter handle_osd_op_reply 20 ondisk uv 7 in 10.1 attempt 0 2023-10-30T20:25:47.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b2c60640 10 client.4367.objecter op 0 rval 0 len 0 2023-10-30T20:25:47.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b2c60640 15 client.4367.objecter handle_osd_op_reply completed tid 20 2023-10-30T20:25:47.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b2c60640 15 client.4367.objecter _finish_op 20 2023-10-30T20:25:47.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b2c60640 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 4 2023-10-30T20:25:47.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b2c60640 15 client.4367.objecter _session_op_remove 0 20 2023-10-30T20:25:47.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b2c60640 5 client.4367.objecter 0 in flight 2023-10-30T20:25:47.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b2c60640 20 librados: complete_aio_write 0x55d109ee8a30 2023-10-30T20:25:47.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1771 ms) 2023-10-30T20:25:47.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b495b780 10 librados: watch_flush enter 2023-10-30T20:25:47.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b495b780 10 librados: watch_flush exit 2023-10-30T20:25:47.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b495b780 10 client.4367.objecter close_session for osd.0 2023-10-30T20:25:47.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 >> v2:172.21.15.69:6801/2068724062 conn(0x55d109e217d0 msgr2=0x55d109e26110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:25:47.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b495b780 1 --2- 172.21.15.69:0/1144027014 >> v2:172.21.15.69:6801/2068724062 conn(0x55d109e217d0 0x55d109e26110 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:25:47.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b495b780 20 client.4367.objecter put_session s=0x55d109e32a00 osd=0 1 2023-10-30T20:25:47.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b495b780 20 client.4367.objecter shutdown clearing up homeless session... 2023-10-30T20:25:47.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b495b780 10 client.4367.objecter successfully canceled tick 2023-10-30T20:25:47.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.755+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa19406aa00 msgr2=0x7fa19406cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:25:47.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.756+0000 7fa1b495b780 1 --2- 172.21.15.69:0/1144027014 >> v2:172.21.15.69:6800/1876307353 conn(0x7fa19406aa00 0x7fa19406cec0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4005770 tx=0x7fa1a40056e0 comp rx=0 tx=0).stop 2023-10-30T20:25:47.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.756+0000 7fa1b495b780 10 monclient: shutdown 2023-10-30T20:25:47.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.756+0000 7fa1b495b780 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:25:47.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.756+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d109da0a30 msgr2=0x55d109e2d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:25:47.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.756+0000 7fa1b495b780 1 --2- 172.21.15.69:0/1144027014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d109da0a30 0x55d109e2d820 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55d109da06e0 tx=0x7fa1a8002fb0 comp rx=0 tx=0).stop 2023-10-30T20:25:47.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.756+0000 7fa1b2c60640 1 -- 172.21.15.69:0/1144027014 reap_dead start 2023-10-30T20:25:47.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.756+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 shutdown_connections 2023-10-30T20:25:47.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.756+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 >> 172.21.15.69:0/1144027014 conn(0x55d109da5890 msgr2=0x55d109e2def0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:25:47.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.756+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 shutdown_connections 2023-10-30T20:25:47.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.756+0000 7fa1b495b780 1 -- 172.21.15.69:0/1144027014 wait complete. 2023-10-30T20:25:47.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.756+0000 7fa1b495b780 1 librados: shutdown 2023-10-30T20:25:47.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] 12 tests from AsioRados (19206 ms total) 2023-10-30T20:25:47.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-10-30T20:25:47.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] Global test environment tear-down 2023-10-30T20:25:47.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (30984 ms total) 2023-10-30T20:25:47.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ PASSED ] 12 tests. 2023-10-30T20:25:47.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2023-10-30T20:25:47.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] Global test environment set-up. 2023-10-30T20:25:47.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2023-10-30T20:25:47.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2023-10-30T20:25:47.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5290 ms) 2023-10-30T20:25:47.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2023-10-30T20:25:47.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (316 ms) 2023-10-30T20:25:47.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2023-10-30T20:25:47.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20117 ms) 2023-10-30T20:25:47.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2023-10-30T20:25:47.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: attempt 0 of 20 2023-10-30T20:25:47.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5172 ms) 2023-10-30T20:25:47.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30895 ms total) 2023-10-30T20:25:47.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: 2023-10-30T20:25:47.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] Global test environment tear-down 2023-10-30T20:25:47.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30896 ms total) 2023-10-30T20:25:47.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ PASSED ] 4 tests. 2023-10-30T20:25:49.107 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2023-10-30T20:25:49.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] Global test environment set-up. 2023-10-30T20:25:49.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2023-10-30T20:25:49.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2023-10-30T20:25:49.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2023-10-30T20:25:49.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-10-30T20:25:49.109 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-10-30T20:25:49.109 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2023-10-30T20:25:49.109 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2023-10-30T20:25:49.109 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: unable to get monitor info from DNS SRV with service name: 2023-10-30T20:25:17.049+0000 7fea9b974780 -1 failed for service _ceph-mon._tcp 2023-10-30T20:25:49.109 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: ceph-mon 2023-10-30T20:25:49.110 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-10-30T20:25:17.049+0000 7fea9b974780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-10-30T20:25:49.110 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (304 ms) 2023-10-30T20:25:49.110 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2023-10-30T20:25:49.110 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2005 ms) 2023-10-30T20:25:49.110 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2309 ms total) 2023-10-30T20:25:49.111 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-10-30T20:25:49.111 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2023-10-30T20:25:49.111 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2023-10-30T20:25:49.111 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e6782810 2023-10-30T20:25:49.111 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e6731e70 2023-10-30T20:25:49.111 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5b33ca0 2023-10-30T20:25:49.112 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e673b2d0 2023-10-30T20:25:49.112 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5e9c940 2023-10-30T20:25:49.112 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5e9d330 2023-10-30T20:25:49.112 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5e9dd70 2023-10-30T20:25:49.112 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5e9e7a0 2023-10-30T20:25:49.113 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5e9f1b0 2023-10-30T20:25:49.113 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5d6a020 2023-10-30T20:25:49.113 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5e94ca0 2023-10-30T20:25:49.113 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5dfa560 2023-10-30T20:25:49.113 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5e91800 2023-10-30T20:25:49.114 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5ea0e10 2023-10-30T20:25:49.114 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5e9ef30 2023-10-30T20:25:49.114 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5ea1ca0 2023-10-30T20:25:49.114 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5ea0be0 2023-10-30T20:25:49.114 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5eab360 2023-10-30T20:25:49.114 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5ea1a50 2023-10-30T20:25:49.115 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5eac260 2023-10-30T20:25:49.115 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5eab160 2023-10-30T20:25:49.115 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5ead130 2023-10-30T20:25:49.115 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5eabf10 2023-10-30T20:25:49.115 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5626e5ea2790 2023-10-30T20:25:49.115 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 24 aios 2023-10-30T20:25:49.116 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e6782810 2023-10-30T20:25:49.116 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e6731e70 2023-10-30T20:25:49.116 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5b33ca0 2023-10-30T20:25:49.116 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e673b2d0 2023-10-30T20:25:49.116 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5e9c940 2023-10-30T20:25:49.117 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5e9d330 2023-10-30T20:25:49.117 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5e9dd70 2023-10-30T20:25:49.117 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5e9e7a0 2023-10-30T20:25:49.117 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5e9f1b0 2023-10-30T20:25:49.117 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5d6a020 2023-10-30T20:25:49.117 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5e94ca0 2023-10-30T20:25:49.118 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5dfa560 2023-10-30T20:25:49.118 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5e91800 2023-10-30T20:25:49.118 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5ea0e10 2023-10-30T20:25:49.118 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5e9ef30 2023-10-30T20:25:49.118 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5ea1ca0 2023-10-30T20:25:49.118 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5ea0be0 2023-10-30T20:25:49.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5eab360 2023-10-30T20:25:49.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5ea1a50 2023-10-30T20:25:49.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5eac260 2023-10-30T20:25:49.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5eab160 2023-10-30T20:25:49.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5ead130 2023-10-30T20:25:49.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5eabf10 2023-10-30T20:25:49.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5626e5ea2790 2023-10-30T20:25:49.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: done. 2023-10-30T20:25:49.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (30052 ms) 2023-10-30T20:25:49.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_ api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2023-10-30T20:25:49.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2023-10-30T20:25:49.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2023-10-30T20:25:49.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2023-10-30T20:25:49.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (32129 ms) 2023-10-30T20:25:49.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (32129 ms total) 2023-10-30T20:25:49.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: 2023-10-30T20:25:49.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2023-10-30T20:25:49.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (32129 ms total) 2023-10-30T20:25:49.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2023-10-30T20:25:49.124 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:11.181 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:49.205 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:11.263 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:49.289 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:11.316 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:49.296 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:11.323 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:49.339 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:11.398 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:49.348 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:11.407 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:49.373 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:11.432 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:49.432 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:11.459 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:49.494 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:11.553 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:25:50.264 INFO:tasks.ceph.osd.4.smithi183.stderr:2023-10-30T20:25:50.257+0000 ea57640 -1 osd.4 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4453.0:2 14.3 14:c6b5651d:test-rados-api-smithi069-91800-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-10-30T20:25:50.695 INFO:tasks.ceph.osd.6.smithi183.stderr:2023-10-30T20:25:50.574+0000 f258640 -1 osd.6 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4418.0:2 16.1a 16:5c6776db:::0.obj:head [write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-10-30T20:25:51.686 INFO:tasks.ceph.osd.6.smithi183.stderr:2023-10-30T20:25:51.682+0000 f258640 -1 osd.6 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4418.0:2 16.1a 16:5c6776db:::0.obj:head [write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-10-30T20:25:52.465 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-10-30T20:25:52.257+0000 1025a640 -1 osd.2 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4459.0:2 21.1b 21:d9e086c3:test-rados-api-smithi069-91909-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-10-30T20:25:52.928 INFO:tasks.ceph.osd.6.smithi183.stderr:2023-10-30T20:25:52.889+0000 f258640 -1 osd.6 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4418.0:2 16.1a 16:5c6776db:::0.obj:head [write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-10-30T20:25:55.496 INFO:tasks.ceph.osd.6.smithi183.stderr:2023-10-30T20:25:55.491+0000 f258640 -1 osd.6 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4418.0:2 16.1a 16:5c6776db:::0.obj:head [write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-10-30T20:26:03.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: Running main() from gmock_main.cc 2023-10-30T20:26:03.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2023-10-30T20:26:03.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] Global test environment set-up. 2023-10-30T20:26:03.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] 9 tests from LibRadosPools 2023-10-30T20:26:03.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2023-10-30T20:26:03.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolList (5708 ms) 2023-10-30T20:26:03.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2023-10-30T20:26:03.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3611 ms) 2023-10-30T20:26:03.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2023-10-30T20:26:03.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3814 ms) 2023-10-30T20:26:03.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2023-10-30T20:26:03.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3787 ms) 2023-10-30T20:26:03.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2023-10-30T20:26:03.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (4977 ms) 2023-10-30T20:26:03.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2023-10-30T20:26:03.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5524 ms) 2023-10-30T20:26:03.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2023-10-30T20:26:03.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5890 ms) 2023-10-30T20:26:03.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2023-10-30T20:26:03.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5101 ms) 2023-10-30T20:26:03.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2023-10-30T20:26:03.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8360 ms) 2023-10-30T20:26:03.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] 9 tests from LibRadosPools (46772 ms total) 2023-10-30T20:26:03.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: 2023-10-30T20:26:03.568 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] Global test environment tear-down 2023-10-30T20:26:03.568 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (46773 ms total) 2023-10-30T20:26:03.568 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ PASSED ] 9 tests. 2023-10-30T20:26:09.456 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:31.515 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:09.461 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:31.520 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:09.523 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:31.550 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:09.526 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:31.553 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:09.535 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:31.594 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:09.539 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:31.598 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:09.585 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:31.644 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:09.605 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:31.632 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:09.642 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:31.701 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:26.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2023-10-30T20:26:26.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2023-10-30T20:26:26.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2023-10-30T20:26:26.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2023-10-30T20:26:26.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2023-10-30T20:26:26.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2023-10-30T20:26:26.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (18208 ms) 2023-10-30T20:26:26.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2023-10-30T20:26:26.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (8801 ms) 2023-10-30T20:26:26.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (27009 ms total) 2023-10-30T20:26:26.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: 2023-10-30T20:26:26.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2023-10-30T20:26:26.821 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2023-10-30T20:26:26.821 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2023-10-30T20:26:26.821 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (8100 ms) 2023-10-30T20:26:26.821 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2023-10-30T20:26:26.821 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2023-10-30T20:26:26.821 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (9369 ms) 2023-10-30T20:26:26.822 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2023-10-30T20:26:26.822 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (2923 ms) 2023-10-30T20:26:26.822 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2023-10-30T20:26:26.822 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (6937 ms) 2023-10-30T20:26:26.822 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2023-10-30T20:26:26.823 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94878155438592 notify_id 330712481794 notifier_gid 4571 2023-10-30T20:26:26.823 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (1517 ms) 2023-10-30T20:26:26.823 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2023-10-30T20:26:26.823 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94878155438592 notify_id 330712481795 notifier_gid 4571 2023-10-30T20:26:26.823 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (747 ms) 2023-10-30T20:26:26.823 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2023-10-30T20:26:26.824 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94878157857472 notify_id 330712481793 notifier_gid 4571 2023-10-30T20:26:26.824 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (384 ms) 2023-10-30T20:26:26.824 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2023-10-30T20:26:26.824 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94878157798096 notify_id 330712481793 notifier_gid 4571 2023-10-30T20:26:26.824 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (309 ms) 2023-10-30T20:26:26.824 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2023-10-30T20:26:26.825 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94878157770288 notify_id 330712481794 notifier_gid 4571 2023-10-30T20:26:26.825 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (233 ms) 2023-10-30T20:26:26.825 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2023-10-30T20:26:26.825 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94878157770288 notify_id 330712481794 notifier_gid 4571 2023-10-30T20:26:26.825 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (262 ms) 2023-10-30T20:26:26.825 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2023-10-30T20:26:26.826 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: trying... 2023-10-30T20:26:26.826 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94878157770288 notify_id 330712481796 notifier_gid 4571 2023-10-30T20:26:26.826 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: timed out 2023-10-30T20:26:32.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flush api_c_read_operations: Running main() from gmock_main.cc 2023-10-30T20:26:32.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2023-10-30T20:26:32.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] Global test environment set-up. 2023-10-30T20:26:32.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2023-10-30T20:26:32.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2023-10-30T20:26:32.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2023-10-30T20:26:32.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2023-10-30T20:26:32.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (18019 ms) 2023-10-30T20:26:32.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2023-10-30T20:26:32.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (2652 ms) 2023-10-30T20:26:32.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2023-10-30T20:26:32.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (3531 ms) 2023-10-30T20:26:32.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2023-10-30T20:26:32.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (21314 ms) 2023-10-30T20:26:32.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2023-10-30T20:26:32.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (7759 ms) 2023-10-30T20:26:32.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2023-10-30T20:26:32.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (5496 ms) 2023-10-30T20:26:32.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2023-10-30T20:26:32.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (1109 ms) 2023-10-30T20:26:32.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2023-10-30T20:26:32.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (458 ms) 2023-10-30T20:26:32.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2023-10-30T20:26:32.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (766 ms) 2023-10-30T20:26:32.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2023-10-30T20:26:32.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (745 ms) 2023-10-30T20:26:32.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2023-10-30T20:26:32.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (137 ms) 2023-10-30T20:26:32.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2023-10-30T20:26:32.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (104 ms) 2023-10-30T20:26:32.969 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2023-10-30T20:26:32.969 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (602 ms) 2023-10-30T20:26:32.969 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2023-10-30T20:26:32.969 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (812 ms) 2023-10-30T20:26:32.969 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2023-10-30T20:26:32.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (505 ms) 2023-10-30T20:26:32.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2023-10-30T20:26:32.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (2230 ms) 2023-10-30T20:26:32.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (66239 ms total) 2023-10-30T20:26:32.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: 2023-10-30T20:26:32.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] Global test environment tear-down 2023-10-30T20:26:32.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (75959 ms total) 2023-10-30T20:26:32.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2023-10-30T20:26:33.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: Running main() from gmock_main.cc 2023-10-30T20:26:33.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2023-10-30T20:26:33.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] Global test environment set-up. 2023-10-30T20:26:33.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2023-10-30T20:26:33.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2023-10-30T20:26:33.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (875 ms) 2023-10-30T20:26:33.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2023-10-30T20:26:33.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (72 ms) 2023-10-30T20:26:33.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2023-10-30T20:26:33.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (75532 ms) 2023-10-30T20:26:33.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (76479 ms total) 2023-10-30T20:26:33.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: 2023-10-30T20:26:33.376 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] Global test environment tear-down 2023-10-30T20:26:33.376 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (76479 ms total) 2023-10-30T20:26:33.376 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2023-10-30T20:26:40.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: Running main() from gmock_main.cc 2023-10-30T20:26:40.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2023-10-30T20:26:40.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] Global test environment set-up. 2023-10-30T20:26:40.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2023-10-30T20:26:40.018 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: seed 92013 2023-10-30T20:26:40.018 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2023-10-30T20:26:40.018 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (26880 ms) 2023-10-30T20:26:40.018 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2023-10-30T20:26:40.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (279 ms) 2023-10-30T20:26:40.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2023-10-30T20:26:40.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (17 ms) 2023-10-30T20:26:40.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2023-10-30T20:26:40.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (3643 ms) 2023-10-30T20:26:40.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2023-10-30T20:26:40.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (13837 ms) 2023-10-30T20:26:40.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (44656 ms total) 2023-10-30T20:26:40.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: 2023-10-30T20:26:40.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2023-10-30T20:26:40.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2023-10-30T20:26:40.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (6906 ms) 2023-10-30T20:26:40.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2023-10-30T20:26:40.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (59 ms) 2023-10-30T20:26:40.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2023-10-30T20:26:40.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (3539 ms) 2023-10-30T20:26:40.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2023-10-30T20:26:40.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (2246 ms) 2023-10-30T20:26:40.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (12750 ms total) 2023-10-30T20:26:40.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: 2023-10-30T20:26:40.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] Global test environment tear-down 2023-10-30T20:26:40.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (83145 ms total) 2023-10-30T20:26:40.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ PASSED ] 9 tests. 2023-10-30T20:26:45.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: Running main() from gmock_main.cc 2023-10-30T20:26:45.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2023-10-30T20:26:45.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] Global test environment set-up. 2023-10-30T20:26:45.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2023-10-30T20:26:45.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2023-10-30T20:26:45.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (747 ms) 2023-10-30T20:26:45.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2023-10-30T20:26:45.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (85 ms) 2023-10-30T20:26:45.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2023-10-30T20:26:45.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (71778 ms) 2023-10-30T20:26:45.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2023-10-30T20:26:45.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:24.457457+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-91800-4", "overlaypool": "test-rados-api-smithi069-91800-4-cache"}]': finished 2023-10-30T20:26:45.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:26.097986+0000 mon.b [INF] from='client.? 172.21.15.69:0/1389976764' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-91800-4-cache", "mode": "writeback"}]: dispatch 2023-10-30T20:26:45.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:28.082629+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-91800-4-cache", "mode": "writeback"}]: dispatch 2023-10-30T20:26:45.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:28.662965+0000 mon.a [WRN] Health check update: 2 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-10-30T20:26:45.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:28.780370+0000 mon.a [INF] from='client.? 172.21.15.69:0/803457257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-91688-10"}]: dispatch 2023-10-30T20:26:45.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:29.297794+0000 mon.a [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-30T20:26:45.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:30.346021+0000 mon.a [INF] from='client.? 172.21.15.69:0/3623643042' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-92013-7"}]: dispatch 2023-10-30T20:26:45.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:31.426978+0000 mon.b [INF] from='client.? 172.21.15.69:0/990297190' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-10-30T20:26:45.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:31.677942+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-91800-4-cache", "mode": "writeback"}]': finished 2023-10-30T20:26:45.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:32.031868+0000 mon.c [INF] from='client.? 172.21.15.69:0/3398435884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-91658-10"}]: dispatch 2023-10-30T20:26:45.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:32.327100+0000 mon.a [INF] from='client.? 172.21.15.69:0/803457257' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-91688-10"}]: dispatch 2023-10-30T20:26:45.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:32.940090+0000 client.admin [INF] onexx 2023-10-30T20:26:45.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:32.961811+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-10-30T20:26:45.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:33.175877+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-91658-10"}]: dispatch 2023-10-30T20:26:45.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:33.507130+0000 mon.a [INF] from='client.? 172.21.15.69:0/803457257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-91688-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T20:26:45.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:35.921842+0000 mon.c [INF] from='client.? 172.21.15.69:0/3398435884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-91658-10"}]: dispatch 2023-10-30T20:26:45.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:36.096092+0000 mon.c [INF] from='client.? 172.21.15.69:0/3383623921' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-91909-7"}]: dispatch 2023-10-30T20:26:45.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:36.771730+0000 mon.c [INF] from='client.? 172.21.15.69:0/3379517387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-91553-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T20:26:45.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:36.779705+0000 mon.c [INF] from='client.? 172.21.15.69:0/1491681851' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-91565-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T20:26:45.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:36.939003+0000 mon.b [INF] from='client.? 172.21.15.69:0/3504146743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-91589-16"}]: dispatch 2023-10-30T20:26:45.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:37.303986+0000 mon.b [INF] from='client.? 172.21.15.69:0/990297190' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-10-30T20:26:45.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:37.442319+0000 mon.b [INF] from='client.? 172.21.15.69:0/1389976764' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-91800-4"}]: dispatch 2023-10-30T20:26:45.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:37.606434+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-91658-10"}]: dispatch 2023-10-30T20:26:45.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:37.611792+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-91909-7"}]: dispatch 2023-10-30T20:26:45.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:37.623459+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-91553-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T20:26:45.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:37.632316+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-91565-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T20:26:45.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:37.639081+0000 mon.a [INF] from='client.? 172.21.15.69:0/803457257' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-91688-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T20:26:45.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:37.640702+0000 mon.a [INF] from='client.? 172.21.15.69:0/3623643042' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-92013-7"}]': finished 2023-10-30T20:26:45.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:38.655162+0000 mon.a [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-30T20:26:45.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:38.661361+0000 mon.a [INF] from='client.? 172.21.15.69:0/803457257' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-91688-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-91688-10"}]: dispatch 2023-10-30T20:26:45.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:38.893801+0000 mon.a [INF] from='client.? 172.21.15.69:0/786261056' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-92176-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T20:26:45.798 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:39.175646+0000 client.admin [INF] twoxx 2023-10-30T20:26:46.884 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: go api_list: [==========] Running 11 tests from 3 test suites. 2023-10-30T20:26:46.885 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] Global test environment set-up. 2023-10-30T20:26:46.885 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 7 tests from LibRadosList 2023-10-30T20:26:46.885 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2023-10-30T20:26:46.885 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjects (63205 ms) 2023-10-30T20:26:46.886 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2023-10-30T20:26:46.886 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (1430 ms) 2023-10-30T20:26:46.886 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2023-10-30T20:26:46.886 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo2,foo3 2023-10-30T20:26:46.886 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2023-10-30T20:26:46.887 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo2 2023-10-30T20:26:46.887 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo3 2023-10-30T20:26:46.887 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo4,foo5 2023-10-30T20:26:46.887 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo4 2023-10-30T20:26:46.887 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo5 2023-10-30T20:26:46.887 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2023-10-30T20:26:46.888 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo6,foo7 2023-10-30T20:26:46.888 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo7 2023-10-30T20:26:46.888 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo6 2023-10-30T20:26:46.888 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-10-30T20:26:46.888 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo4 2023-10-30T20:26:46.889 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo5 2023-10-30T20:26:46.889 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo7 2023-10-30T20:26:46.889 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo6 2023-10-30T20:26:46.889 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo1 2023-10-30T20:26:46.889 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo1 2023-10-30T20:26:46.890 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo2 2023-10-30T20:26:46.890 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo3 2023-10-30T20:26:46.890 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (7954 ms) 2023-10-30T20:26:46.890 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2023-10-30T20:26:46.890 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 1 0 2023-10-30T20:26:46.891 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 10 0 2023-10-30T20:26:46.891 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 13 0 2023-10-30T20:26:46.891 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 7 0 2023-10-30T20:26:46.891 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 14 0 2023-10-30T20:26:46.891 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 0 0 2023-10-30T20:26:46.891 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 15 0 2023-10-30T20:26:46.892 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 11 0 2023-10-30T20:26:46.892 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 5 0 2023-10-30T20:26:46.892 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 8 0 2023-10-30T20:26:46.892 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 6 0 2023-10-30T20:26:46.892 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 3 0 2023-10-30T20:26:46.892 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 4 0 2023-10-30T20:26:46.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 12 0 2023-10-30T20:26:46.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 9 0 2023-10-30T20:26:46.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2 0 2023-10-30T20:26:46.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-10-30T20:26:46.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (8243 ms) 2023-10-30T20:26:46.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2023-10-30T20:26:46.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: x cursor=MIN 2023-10-30T20:26:46.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=1 cursor=8:02547ec2:::1:head 2023-10-30T20:26:46.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=10 cursor=8:52ea6a34:::10:head 2023-10-30T20:26:46.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=13 cursor=8:566253c9:::13:head 2023-10-30T20:26:46.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=7 cursor=8:5c6b0b28:::7:head 2023-10-30T20:26:46.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=14 cursor=8:62a1935d:::14:head 2023-10-30T20:26:46.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=0 cursor=8:6cac518f:::0:head 2023-10-30T20:26:46.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=15 cursor=8:863748b0:::15:head 2023-10-30T20:26:46.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=11 cursor=8:89d3ae78:::11:head 2023-10-30T20:26:46.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=5 cursor=8:b29083e3:::5:head 2023-10-30T20:26:46.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=8 cursor=8:bd63b0f1:::8:head 2023-10-30T20:26:46.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=6 cursor=8:c4fdafeb:::6:head 2023-10-30T20:26:46.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=3 cursor=8:cfc208b3:::3:head 2023-10-30T20:26:46.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=4 cursor=8:d83876eb:::4:head 2023-10-30T20:26:46.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=12 cursor=8:de5d7c5f:::12:head 2023-10-30T20:26:46.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=9 cursor=8:e960b815:::9:head 2023-10-30T20:26:46.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=2 cursor=8:f905c69b:::2:head 2023-10-30T20:26:46.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: FIRST> seek to MIN oid=1 2023-10-30T20:26:46.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=1 cursor=8:02547ec2:::1:head 2023-10-30T20:26:46.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:02547ec2:::1:head 2023-10-30T20:26:46.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:02547ec2:::1:head -> 1 2023-10-30T20:26:46.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=10 cursor=8:52ea6a34:::10:head 2023-10-30T20:26:46.898 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:52ea6a34:::10:head 2023-10-30T20:26:46.898 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2023-10-30T20:26:46.898 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=13 cursor=8:566253c9:::13:head 2023-10-30T20:26:46.898 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:566253c9:::13:head 2023-10-30T20:26:46.898 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:566253c9:::13:head -> 13 2023-10-30T20:26:46.898 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=7 cursor=8:5c6b0b28:::7:head 2023-10-30T20:26:46.899 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2023-10-30T20:26:46.899 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2023-10-30T20:26:47.359 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:09.417 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:47.418 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:09.476 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:47.980 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:10.039 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:48.554 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:10.613 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:48.603 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:04:10.630 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:48.658 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:04:10.685 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:48.720 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:10.779 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:48.783 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:04:10.810 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:48.892 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:10.951 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:26:49.699 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=14 cursor=8:62a1935d:::14:head 2023-10-30T20:26:49.699 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:62a1935d:::14:head 2023-10-30T20:26:49.699 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:62a1935d:::14:head -> 14 2023-10-30T20:26:49.700 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=0 cursor=8:6cac518f:::0:head 2023-10-30T20:26:49.700 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:6cac518f:::0:head 2023-10-30T20:26:49.700 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:6cac518f:::0:head -> 0 2023-10-30T20:26:49.701 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=15 cursor=8:863748b0:::15:head 2023-10-30T20:26:49.701 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:863748b0:::15:head 2023-10-30T20:26:49.701 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:863748b0:::15:head -> 15 2023-10-30T20:26:49.701 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=11 cursor=8:89d3ae78:::11:head 2023-10-30T20:26:49.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:89d3ae78:::11:head 2023-10-30T20:26:49.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2023-10-30T20:26:49.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=5 cursor=8:b29083e3:::5:head 2023-10-30T20:26:49.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:b29083e3:::5:head 2023-10-30T20:26:49.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:b29083e3:::5:head -> 5 2023-10-30T20:26:49.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=8 cursor=8:bd63b0f1:::8:head 2023-10-30T20:26:49.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2023-10-30T20:26:49.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2023-10-30T20:26:49.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=6 cursor=8:c4fdafeb:::6:head 2023-10-30T20:26:49.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2023-10-30T20:26:49.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2023-10-30T20:26:49.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=3 cursor=8:cfc208b3:::3:head 2023-10-30T20:26:49.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:cfc208b3:::3:head 2023-10-30T20:26:49.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2023-10-30T20:26:49.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=4 cursor=8:d83876eb:::4:head 2023-10-30T20:26:49.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:d83876eb:::4:head 2023-10-30T20:26:49.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:d83876eb:::4:head -> 4 2023-10-30T20:26:49.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=12 cursor=8:de5d7c5f:::12:head 2023-10-30T20:26:49.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2023-10-30T20:26:49.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2023-10-30T20:26:49.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=9 cursor=8:e960b815:::9:head 2023-10-30T20:26:49.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:e960b815:::9:head 2023-10-30T20:26:49.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:e960b815:::9:head -> 9 2023-10-30T20:26:49.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=2 cursor=8:f905c69b:::2:head 2023-10-30T20:26:49.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:f905c69b:::2:head 2023-10-30T20:26:49.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:f905c69b:::2:head -> 2 2023-10-30T20:26:49.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:52ea6a34:::10:head 2023-10-30T20:26:49.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:52ea6a34:::10:head expected=8:52ea6a34:::10:head 2023-10-30T20:26:49.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2023-10-30T20:26:49.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=10 expected=10 2023-10-30T20:26:49.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2023-10-30T20:26:49.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:5c6b0b28:::7:head expected=8:5c6b0b28:::7:head 2023-10-30T20:26:49.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2023-10-30T20:26:49.711 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=7 expected=7 2023-10-30T20:26:49.711 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:cfc208b3:::3:head 2023-10-30T20:26:49.711 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:cfc208b3:::3:head expected=8:cfc208b3:::3:head 2023-10-30T20:26:49.711 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2023-10-30T20:26:49.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=3 expected=3 2023-10-30T20:26:49.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2023-10-30T20:26:49.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:c4fdafeb:::6:head expected=8:c4fdafeb:::6:head 2023-10-30T20:26:49.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2023-10-30T20:26:49.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=6 expected=6 2023-10-30T20:26:49.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2023-10-30T20:26:49.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:bd63b0f1:::8:head expected=8:bd63b0f1:::8:head 2023-10-30T20:26:49.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2023-10-30T20:26:49.714 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=8 expected=8 2023-10-30T20:26:49.714 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:b29083e3:::5:head 2023-10-30T20:26:49.714 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:b29083e3:::5:head expected=8:b29083e3:::5:head 2023-10-30T20:26:49.714 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:b29083e3:::5:head -> 5 2023-10-30T20:26:49.715 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=5 expected=5 2023-10-30T20:26:49.715 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:02547ec2:::1:head 2023-10-30T20:26:49.715 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:02547ec2:::1:head expected=8:02547ec2:::1:head 2023-10-30T20:26:49.715 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:02547ec2:::1:head -> 1 2023-10-30T20:26:49.716 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=1 expected=1 2023-10-30T20:26:49.716 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:f905c69b:::2:head 2023-10-30T20:26:49.716 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:f905c69b:::2:head expected=8:f905c69b:::2:head 2023-10-30T20:26:49.716 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:f905c69b:::2:head -> 2 2023-10-30T20:26:49.717 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=2 expected=2 2023-10-30T20:26:49.717 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:e960b815:::9:head 2023-10-30T20:26:50.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: t: 2023-10-30T20:26:39.571791+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-91589-16"}]: dispatch 2023-10-30T20:26:50.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:39.616756+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-10-30T20:26:50.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:39.627339+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-91800-4"}]: dispatch 2023-10-30T20:26:50.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:38.711622+0000 mon.c [INF] from='client.? 172.21.15.69:0/1124164501' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-92076-4"}]: dispatch 2023-10-30T20:26:50.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:39.018357+0000 mon.c [INF] from='client.? 172.21.15.69:0/3398435884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-91658-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T20:26:50.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:39.189889+0000 mon.c [INF] from='client.? 172.21.15.69:0/3383623921' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-91909-7"}]: dispatch 2023-10-30T20:26:50.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:39.204584+0000 mon.b [INF] from='client.? 172.21.15.69:0/1450406258' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-91608-22"}]: dispatch 2023-10-30T20:26:50.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:40.629927+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-91608-22"}]: dispatch 2023-10-30T20:26:50.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:40.841088+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-92076-4"}]: dispatch 2023-10-30T20:26:50.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:40.853560+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-91658-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T20:26:50.611 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:40.857464+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-91909-7"}]: dispatch 2023-10-30T20:26:50.611 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:40.881303+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-91553-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T20:26:50.611 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:40.886570+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-91565-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T20:26:50.611 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:42.290524+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-10-30T20:26:50.611 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:42.000174+0000 mon.b [INF] from='client.? 172.21.15.69:0/1450406258' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-91608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T20:26:50.611 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:42.062379+0000 mon.c [INF] from='client.? 172.21.15.69:0/3383623921' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-91909-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T20:26:54.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:42.272657+0000 mon.b [INF] from='client.? 172.21.15.69:0/3504146743' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-91589-16"}]: dispatch 2023-10-30T20:26:54.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:43.444970+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-91608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T20:26:54.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:43.453901+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-91589-16"}]: dispatch 2023-10-30T20:26:54.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:43.868330+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-91909-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T20:26:54.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:43.912908+0000 mon.a [INF] from='client.? 172.21.15.69:0/786261056' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-92176-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T20:26:54.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:43.914484+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-91800-4"}]': finished 2023-10-30T20:26:54.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:43.916760+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-92076-4"}]': finished 2023-10-30T20:26:54.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:43.919339+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-91658-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T20:26:54.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:45.552929+0000 mon.c [INF] from='client.? 172.21.15.69:0/1124164501' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-92076-4", "tierpool": "test-rados-api-smithi069-92076-6"}]: dispatch 2023-10-30T20:26:54.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:45.583440+0000 mon.b [INF] from='client.? 172.21.15.69:0/990297190' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-10-30T20:26:54.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:45.593668+0000 mon.c [INF] from='client.? 172.21.15.69:0/3398435884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-91658-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-91658-10"}]: dispatch 2023-10-30T20:26:54.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:45.770254+0000 mon.b [INF] from='client.? 172.21.15.69:0/3504146743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-91589-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T20:26:54.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:45.778804+0000 mon.b [INF] from='client.? 172.21.15.69:0/1389976764' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-91800-4", "tierpool": "test-rados-api-smithi069-91800-4-cache"}]: dispatch 2023-10-30T20:26:54.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:47.001077+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-92076-4", "tierpool": "test-rados-api-smithi069-92076-6"}]: dispatch 2023-10-30T20:26:54.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:47.005693+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-91658-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-91658-10"}]: dispatch 2023-10-30T20:26:54.068 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:47.242767+0000 client.admin [INF] threexx 2023-10-30T20:27:02.399 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:47.310256+0000 mon.a [INF] from='client. list_parallel: process_1_[92288]: starting. 2023-10-30T20:27:02.400 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[92288]: creating pool ceph_test_rados_list_parallel.smithi069-92256 2023-10-30T20:27:02.400 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[92288]: created object 0... 2023-10-30T20:27:02.400 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[92288]: created object 25... 2023-10-30T20:27:02.400 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[92288]: created object 49... 2023-10-30T20:27:02.400 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[92288]: finishing. 2023-10-30T20:27:02.401 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[92288]: shutting down. 2023-10-30T20:27:02.401 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[92290]: starting. 2023-10-30T20:27:02.401 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[92290]: listing objects. 2023-10-30T20:27:02.401 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[92290]: listed object 0... 2023-10-30T20:27:02.401 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[92290]: listed object 25... 2023-10-30T20:27:02.402 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[92290]: saw 50 objects 2023-10-30T20:27:02.402 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[92290]: shutting down. 2023-10-30T20:27:02.402 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:02.402 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[105957]: starting. 2023-10-30T20:27:02.402 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[105957]: creating pool ceph_test_rados_list_parallel.smithi069-92256 2023-10-30T20:27:02.403 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[105957]: created object 0... 2023-10-30T20:27:02.403 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[105957]: created object 25... 2023-10-30T20:27:02.403 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[105957]: created object 49... 2023-10-30T20:27:02.403 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[105957]: finishing. 2023-10-30T20:27:02.403 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[105957]: shutting down. 2023-10-30T20:27:02.403 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:02.404 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[105958]: starting. 2023-10-30T20:27:02.404 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[105958]: listing objects. 2023-10-30T20:27:02.404 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[105958]: listed object 0... 2023-10-30T20:27:02.404 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[105958]: listed object 25... 2023-10-30T20:27:02.404 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[105958]: saw 45 objects 2023-10-30T20:27:02.405 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[105958]: shutting down. 2023-10-30T20:27:02.405 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:02.405 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[105959]: starting. 2023-10-30T20:27:02.405 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[105959]: removed 25 objects... 2023-10-30T20:27:02.405 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[105959]: removed half of the objects 2023-10-30T20:27:02.405 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[105959]: removed 50 objects... 2023-10-30T20:27:02.406 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[105959]: removed 50 objects 2023-10-30T20:27:02.406 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[105959]: shutting down. 2023-10-30T20:27:02.406 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:02.406 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:02.406 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[117763]: starting. 2023-10-30T20:27:02.407 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[117763]: creating pool ceph_test_rados_list_parallel.smithi069-92256 2023-10-30T20:27:02.407 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[117763]: created object 0... 2023-10-30T20:27:02.407 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[117763]: created object 25... 2023-10-30T20:27:02.407 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[117763]: created object 49... 2023-10-30T20:27:02.407 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[117763]: finishing. 2023-10-30T20:27:02.407 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[117763]: shutting down. 2023-10-30T20:27:02.408 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:02.408 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:02.408 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[117764]: starting. 2023-10-30T20:27:02.408 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[117764]: listing objects. 2023-10-30T20:27:02.408 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[117764]: listed object 0... 2023-10-30T20:27:02.408 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[117764]: listed object 25... 2023-10-30T20:27:02.409 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[117764]: listed object 50... 2023-10-30T20:27:02.409 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[117764]: saw 55 objects 2023-10-30T20:27:02.409 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[117764]: shutting down. 2023-10-30T20:27:02.409 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:02.409 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:02.409 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[117765]: starting. 2023-10-30T20:27:02.410 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[117765]: added 25 objects... 2023-10-30T20:27:02.410 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[117765]: added half of the objects 2023-10-30T20:27:02.410 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[117765]: added 50 objects... 2023-10-30T20:27:02.410 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[117765]: added 50 objects 2023-10-30T20:27:02.410 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[117765]: shutting down. 2023-10-30T20:27:02.410 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:02.411 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:07.165 INFO:tasks.workunit.client.0.smithi069.stdout: ? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-10-30T20:27:07.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:47.315955+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-91589-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T20:27:07.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:47.321444+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-91800-4", "tierpool": "test-rados-api-smithi069-91800-4-cache"}]: dispatch 2023-10-30T20:27:07.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:47.353420+0000 mon.a [INF] from='client.? 172.21.15.69:0/803457257' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-91688-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-91688-10"}]': finished 2023-10-30T20:27:07.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:47.355350+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-91608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T20:27:07.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:47.360030+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-91909-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T20:27:07.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:48.484403+0000 mon.a [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 2 cache pools are missing hit_sets) 2023-10-30T20:27:07.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-10-30T20:26:48.491742+0000 mon.a [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-30T20:27:07.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (37656 ms) 2023-10-30T20:27:07.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (110266 ms total) 2023-10-30T20:27:07.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: 2023-10-30T20:27:07.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] Global test environment tear-down 2023-10-30T20:27:07.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (110266 ms total) 2023-10-30T20:27:07.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ PASSED ] 4 tests. 2023-10-30T20:27:09.979 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[92381]: starting. 2023-10-30T20:27:09.980 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[92381]: creating pool ceph_test_rados_delete_pools_parallel.smithi069-92350 2023-10-30T20:27:09.980 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[92381]: created object 0... 2023-10-30T20:27:09.980 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[92381]: created object 25... 2023-10-30T20:27:09.980 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[92381]: created object 49... 2023-10-30T20:27:09.981 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[92381]: finishing. 2023-10-30T20:27:09.981 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[92381]: shutting down. 2023-10-30T20:27:09.981 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[92382]: starting. 2023-10-30T20:27:09.981 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[92382]: deleting pool ceph_test_rados_delete_pools_parallel.smithi069-92350 2023-10-30T20:27:09.981 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[92382]: shutting down. 2023-10-30T20:27:09.981 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-10-30T20:27:09.982 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[109321]: starting. 2023-10-30T20:27:09.982 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[109321]: creating pool ceph_test_rados_delete_pools_parallel.smithi069-92350 2023-10-30T20:27:09.982 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[109321]: created object 0... 2023-10-30T20:27:09.982 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[109321]: created object 25... 2023-10-30T20:27:09.982 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[109321]: created object 49... 2023-10-30T20:27:09.983 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[109321]: finishing. 2023-10-30T20:27:09.983 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[109321]: shutting down. 2023-10-30T20:27:09.983 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-10-30T20:27:09.983 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[109323]: starting. 2023-10-30T20:27:09.983 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[109323]: listing objects. 2023-10-30T20:27:09.984 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[109323]: listed object 0... 2023-10-30T20:27:09.984 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[109323]: listed object 25... 2023-10-30T20:27:09.984 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[109323]: saw 50 objects 2023-10-30T20:27:09.984 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[109323]: shutting down. 2023-10-30T20:27:09.984 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-10-30T20:27:09.985 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[109322]: starting. 2023-10-30T20:27:09.985 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[109322]: deleting pool ceph_test_rados_delete_pools_parallel.smithi069-92350 2023-10-30T20:27:09.985 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[109322]: shutting down. 2023-10-30T20:27:09.985 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-10-30T20:27:09.985 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-10-30T20:27:09.985 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******* SUCCESS ********** 2023-10-30T20:27:10.975 INFO:tasks.workunit.client.0.smithi069.stdout:ing 2023-10-30T20:27:10.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushed 2023-10-30T20:27:10.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3147 ms) 2023-10-30T20:27:10.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2023-10-30T20:27:10.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: trying... 2023-10-30T20:27:10.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94878157773408 notify_id 335007449093 notifier_gid 4571 2023-10-30T20:27:10.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: timed out 2023-10-30T20:27:10.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2023-10-30T20:27:10.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushed 2023-10-30T20:27:10.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3163 ms) 2023-10-30T20:27:10.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2023-10-30T20:27:10.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: List watches 2023-10-30T20:27:10.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 2023-10-30T20:27:10.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94878156522272 notify_id 339302416390 notifier_gid 4571 2023-10-30T20:27:10.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 done 2023-10-30T20:27:10.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: watch_check 2023-10-30T20:27:10.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: unwatch2 2023-10-30T20:27:10.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2023-10-30T20:27:10.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: done 2023-10-30T20:27:10.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3529 ms) 2023-10-30T20:27:10.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2023-10-30T20:27:10.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: List watches 2023-10-30T20:27:10.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 2023-10-30T20:27:10.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94878156522272 notify_id 343597383683 notifier_gid 4571 2023-10-30T20:27:10.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 done 2023-10-30T20:27:10.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: watch_check 2023-10-30T20:27:10.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: unwatch2 2023-10-30T20:27:10.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2023-10-30T20:27:10.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: done 2023-10-30T20:27:10.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3557 ms) 2023-10-30T20:27:10.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (44177 ms total) 2023-10-30T20:27:10.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: 2023-10-30T20:27:10.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2023-10-30T20:27:10.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (114083 ms total) 2023-10-30T20:27:10.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2023-10-30T20:27:14.164 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[92358]: starting. 2023-10-30T20:27:14.164 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[92358]: creating pool ceph_test_rados_open_pools_parallel.smithi069-92326 2023-10-30T20:27:14.164 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[92358]: created object 0... 2023-10-30T20:27:14.165 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[92358]: created object 25... 2023-10-30T20:27:14.165 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[92358]: created object 49... 2023-10-30T20:27:14.165 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[92358]: finishing. 2023-10-30T20:27:14.165 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[92358]: shutting down. 2023-10-30T20:27:14.165 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[92363]: starting. 2023-10-30T20:27:14.165 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[92363]: rados_pool_create. 2023-10-30T20:27:14.166 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[92363]: rados_ioctx_create. 2023-10-30T20:27:14.166 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[92363]: shutting down. 2023-10-30T20:27:14.166 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2023-10-30T20:27:14.166 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[112543]: starting. 2023-10-30T20:27:14.166 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[112543]: creating pool ceph_test_rados_open_pools_parallel.smithi069-92326 2023-10-30T20:27:14.166 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[112543]: created object 0... 2023-10-30T20:27:14.167 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[112543]: created object 25... 2023-10-30T20:27:14.167 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[112543]: created object 49... 2023-10-30T20:27:14.167 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[112543]: finishing. 2023-10-30T20:27:14.167 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[112543]: shutting down. 2023-10-30T20:27:14.167 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2023-10-30T20:27:14.167 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[112546]: starting. 2023-10-30T20:27:14.168 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[112546]: rados_pool_create. 2023-10-30T20:27:14.168 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[112546]: rados_ioctx_create. 2023-10-30T20:27:14.168 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[112546]: shutting down. 2023-10-30T20:27:14.168 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2023-10-30T20:27:14.168 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2023-10-30T20:27:14.169 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******* SUCCESS ********** 2023-10-30T20:27:16.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: Running main() from gmock_main.cc 2023-10-30T20:27:16.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2023-10-30T20:27:16.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] Global test environment set-up. 2023-10-30T20:27:16.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2023-10-30T20:27:16.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: seed 91608 2023-10-30T20:27:16.421 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2023-10-30T20:27:16.421 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2023-10-30T20:27:16.421 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2023-10-30T20:27:16.421 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (23898 ms) 2023-10-30T20:27:16.421 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2023-10-30T20:27:16.421 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (1296 ms) 2023-10-30T20:27:16.422 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2023-10-30T20:27:16.422 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (1031 ms) 2023-10-30T20:27:16.422 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2023-10-30T20:27:16.422 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (1270 ms) 2023-10-30T20:27:16.422 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2023-10-30T20:27:16.423 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (7967 ms) 2023-10-30T20:27:16.423 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2023-10-30T20:27:16.423 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (4150 ms) 2023-10-30T20:27:16.423 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2023-10-30T20:27:16.424 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (5943 ms) 2023-10-30T20:27:16.424 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2023-10-30T20:27:16.424 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (5336 ms) 2023-10-30T20:27:16.424 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2023-10-30T20:27:16.424 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (2667 ms) 2023-10-30T20:27:16.425 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2023-10-30T20:27:16.425 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (2761 ms) 2023-10-30T20:27:16.425 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2023-10-30T20:27:16.425 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (2916 ms) 2023-10-30T20:27:16.425 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2023-10-30T20:27:16.426 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (1753 ms) 2023-10-30T20:27:16.426 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2023-10-30T20:27:16.426 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (1885 ms) 2023-10-30T20:27:16.426 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2023-10-30T20:27:16.426 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (145 ms) 2023-10-30T20:27:16.426 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2023-10-30T20:27:16.427 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (700 ms) 2023-10-30T20:27:16.427 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-10-30T20:27:16.427 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (197 ms) 2023-10-30T20:27:16.427 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2023-10-30T20:27:16.427 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (175 ms) 2023-10-30T20:27:16.428 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2023-10-30T20:27:16.428 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (93 ms) 2023-10-30T20:27:16.428 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2023-10-30T20:27:16.428 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (205 ms) 2023-10-30T20:27:16.428 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (64388 ms total) 2023-10-30T20:27:16.428 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: 2023-10-30T20:27:16.429 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2023-10-30T20:27:16.429 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2023-10-30T20:27:16.429 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (6676 ms) 2023-10-30T20:27:16.429 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2023-10-30T20:27:16.429 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (3123 ms) 2023-10-30T20:27:16.430 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2023-10-30T20:27:16.430 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (432 ms) 2023-10-30T20:27:16.430 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2023-10-30T20:27:16.430 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (448 ms) 2023-10-30T20:27:16.430 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2023-10-30T20:27:18.390 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.Roun list_parallel: ******************************* 2023-10-30T20:27:18.390 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[127573]: starting. 2023-10-30T20:27:18.390 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[127573]: creating pool ceph_test_rados_list_parallel.smithi069-92256 2023-10-30T20:27:18.391 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[127573]: created object 0... 2023-10-30T20:27:18.391 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[127573]: created object 25... 2023-10-30T20:27:18.391 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[127573]: created object 49... 2023-10-30T20:27:18.391 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[127573]: finishing. 2023-10-30T20:27:18.391 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[127573]: shutting down. 2023-10-30T20:27:18.392 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.392 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.392 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.392 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[127574]: starting. 2023-10-30T20:27:18.392 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[127574]: listing objects. 2023-10-30T20:27:18.392 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[127574]: listed object 0... 2023-10-30T20:27:18.393 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[127574]: listed object 25... 2023-10-30T20:27:18.393 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[127574]: listed object 50... 2023-10-30T20:27:18.393 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[127574]: listed object 75... 2023-10-30T20:27:18.393 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[127574]: saw 100 objects 2023-10-30T20:27:18.393 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[127574]: shutting down. 2023-10-30T20:27:18.394 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.394 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.394 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.394 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[127577]: starting. 2023-10-30T20:27:18.394 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[127577]: removed 25 objects... 2023-10-30T20:27:18.395 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[127577]: removed half of the objects 2023-10-30T20:27:18.395 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[127577]: removed 50 objects... 2023-10-30T20:27:18.395 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[127577]: removed 50 objects 2023-10-30T20:27:18.395 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[127577]: shutting down. 2023-10-30T20:27:18.395 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.395 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.396 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.396 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[127576]: starting. 2023-10-30T20:27:18.396 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[127576]: added 25 objects... 2023-10-30T20:27:18.396 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[127576]: added half of the objects 2023-10-30T20:27:18.396 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[127576]: added 50 objects... 2023-10-30T20:27:18.396 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[127576]: added 50 objects 2023-10-30T20:27:18.397 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[127576]: shutting down. 2023-10-30T20:27:18.397 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.397 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.397 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.397 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[127575]: starting. 2023-10-30T20:27:18.398 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[127575]: added 25 objects... 2023-10-30T20:27:18.398 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[127575]: added half of the objects 2023-10-30T20:27:18.398 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[127575]: added 50 objects... 2023-10-30T20:27:18.398 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[127575]: added 50 objects 2023-10-30T20:27:18.398 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[127575]: shutting down. 2023-10-30T20:27:18.398 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.399 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.399 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.399 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.399 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[135446]: starting. 2023-10-30T20:27:18.399 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[135446]: creating pool ceph_test_rados_list_parallel.smithi069-92256 2023-10-30T20:27:18.399 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[135446]: created object 0... 2023-10-30T20:27:18.400 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[135446]: created object 25... 2023-10-30T20:27:18.400 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[135446]: created object 49... 2023-10-30T20:27:18.400 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[135446]: finishing. 2023-10-30T20:27:18.400 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[135446]: shutting down. 2023-10-30T20:27:18.400 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.400 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.401 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.401 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:18.401 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135447]: starting. 2023-10-30T20:27:18.401 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135447]: listing objects. 2023-10-30T20:27:24.720 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135447] api_lock: Running main() from gmock_main.cc 2023-10-30T20:27:24.720 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2023-10-30T20:27:24.720 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] Global test environment set-up. 2023-10-30T20:27:24.720 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLock 2023-10-30T20:27:24.721 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2023-10-30T20:27:24.721 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (9425 ms) 2023-10-30T20:27:24.721 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2023-10-30T20:27:24.721 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockShared (28849 ms) 2023-10-30T20:27:24.721 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2023-10-30T20:27:24.721 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (4506 ms) 2023-10-30T20:27:24.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2023-10-30T20:27:24.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (4364 ms) 2023-10-30T20:27:24.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2023-10-30T20:27:24.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (3042 ms) 2023-10-30T20:27:24.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2023-10-30T20:27:24.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.Unlock (2788 ms) 2023-10-30T20:27:24.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2023-10-30T20:27:24.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (6589 ms) 2023-10-30T20:27:24.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2023-10-30T20:27:24.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (1801 ms) 2023-10-30T20:27:24.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLock (61364 ms total) 2023-10-30T20:27:24.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: 2023-10-30T20:27:24.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2023-10-30T20:27:24.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2023-10-30T20:27:24.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (6113 ms) 2023-10-30T20:27:24.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2023-10-30T20:27:24.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (132 ms) 2023-10-30T20:27:24.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2023-10-30T20:27:24.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1232 ms) 2023-10-30T20:27:24.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2023-10-30T20:27:24.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1567 ms) 2023-10-30T20:27:24.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2023-10-30T20:27:24.726 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (248 ms) 2023-10-30T20:27:24.726 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2023-10-30T20:27:24.726 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (254 ms) 2023-10-30T20:27:24.726 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2023-10-30T20:27:24.726 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (447 ms) 2023-10-30T20:27:24.727 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2023-10-30T20:27:24.727 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (198 ms) 2023-10-30T20:27:24.727 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (10191 ms total) 2023-10-30T20:27:24.727 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: 2023-10-30T20:27:24.727 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] Global test environment tear-down 2023-10-30T20:27:24.727 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (127979 ms total) 2023-10-30T20:27:24.728 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ PASSED ] 16 tests. 2023-10-30T20:27:29.742 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: Running main() from gmock_main.cc 2023-10-30T20:27:29.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2023-10-30T20:27:29.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] Global test environment set-up. 2023-10-30T20:27:29.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2023-10-30T20:27:29.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: seed 91658 2023-10-30T20:27:29.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2023-10-30T20:27:29.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (15334 ms) 2023-10-30T20:27:29.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2023-10-30T20:27:29.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (16072 ms) 2023-10-30T20:27:29.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2023-10-30T20:27:29.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (11977 ms) 2023-10-30T20:27:29.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2023-10-30T20:27:29.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (8134 ms) 2023-10-30T20:27:29.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2023-10-30T20:27:29.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (805 ms) 2023-10-30T20:27:29.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2023-10-30T20:27:29.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (3127 ms) 2023-10-30T20:27:29.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2023-10-30T20:27:29.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (4990 ms) 2023-10-30T20:27:29.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2023-10-30T20:27:29.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (932 ms) 2023-10-30T20:27:29.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (61371 ms total) 2023-10-30T20:27:29.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: 2023-10-30T20:27:29.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2023-10-30T20:27:29.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2023-10-30T20:27:29.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (10236 ms) 2023-10-30T20:27:29.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2023-10-30T20:27:29.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (115 ms) 2023-10-30T20:27:29.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2023-10-30T20:27:29.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1170 ms) 2023-10-30T20:27:29.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2023-10-30T20:27:29.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (3302 ms) 2023-10-30T20:27:29.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2023-10-30T20:27:29.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (166 ms) 2023-10-30T20:27:29.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2023-10-30T20:27:29.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (1640 ms) 2023-10-30T20:27:29.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2023-10-30T20:27:29.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (292 ms) 2023-10-30T20:27:29.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2023-10-30T20:27:29.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (721 ms) 2023-10-30T20:27:29.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (17644 ms total) 2023-10-30T20:27:29.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: 2023-10-30T20:27:29.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] Global test environment tear-down 2023-10-30T20:27:29.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (133041 ms total) 2023-10-30T20:27:29.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ PASSED ] 16 tests. 2023-10-30T20:27:29.751 INFO:tasks.workunit.client.0.smithi069.stdout:misc: [----------] 1 test from LibRadosMiscPool (30052 ms total) 2023-10-30T20:27:29.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-10-30T20:27:29.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2023-10-30T20:27:29.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2023-10-30T20:27:29.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2023-10-30T20:27:29.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2023-10-30T20:27:29.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.Exec (5309 ms) 2023-10-30T20:27:29.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2023-10-30T20:27:29.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (9941 ms) 2023-10-30T20:27:29.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2023-10-30T20:27:29.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (3459 ms) 2023-10-30T20:27:29.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2023-10-30T20:27:29.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5480 ms) 2023-10-30T20:27:29.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2023-10-30T20:27:29.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2023-10-30T20:27:29.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2023-10-30T20:27:29.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2023-10-30T20:27:29.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2023-10-30T20:27:29.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (71412 ms) 2023-10-30T20:27:29.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 8 tests from LibRadosMisc (95601 ms total) 2023-10-30T20:27:29.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-10-30T20:27:29.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] Global test environment tear-down 2023-10-30T20:27:29.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (132993 ms total) 2023-10-30T20:27:29.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ PASSED ] 12 tests. 2023-10-30T20:27:30.451 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:52.509 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:27:30.453 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:52.511 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:27:30.519 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:04:52.546 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:27:30.525 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:04:52.552 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:27:30.526 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:52.585 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:27:30.533 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:52.592 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:27:30.577 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: Running main() from gmock_main.cc 2023-10-30T20:27:30.577 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2023-10-30T20:27:30.577 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] Global test environment set-up. 2023-10-30T20:27:30.578 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 14 tests from LibRadosIo 2023-10-30T20:27:30.578 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2023-10-30T20:27:30.578 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (21530 ms) 2023-10-30T20:27:30.578 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2023-10-30T20:27:30.578 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2023-10-30T20:27:30.578 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2023-10-30T20:27:30.579 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: no timeout :/ 2023-10-30T20:27:30.579 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: no timeout :/ 2023-10-30T20:27:30.579 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: no timeout :/ 2023-10-30T20:27:30.579 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: no timeout :/ 2023-10-30T20:27:30.579 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (3794 ms) 2023-10-30T20:27:30.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2023-10-30T20:27:30.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (4510 ms) 2023-10-30T20:27:30.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2023-10-30T20:27:30.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.Checksum (2124 ms) 2023-10-30T20:27:30.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2023-10-30T20:27:30.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (3184 ms) 2023-10-30T20:27:30.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2023-10-30T20:27:30.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (12407 ms) 2023-10-30T20:27:30.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2023-10-30T20:27:30.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (2906 ms) 2023-10-30T20:27:30.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2023-10-30T20:27:30.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2790 ms) 2023-10-30T20:27:30.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2023-10-30T20:27:30.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.TruncTest (7113 ms) 2023-10-30T20:27:30.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2023-10-30T20:27:30.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (2402 ms) 2023-10-30T20:27:30.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2023-10-30T20:27:30.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (216 ms) 2023-10-30T20:27:30.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2023-10-30T20:27:30.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RmXattr (650 ms) 2023-10-30T20:27:30.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2023-10-30T20:27:30.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.XattrIter (356 ms) 2023-10-30T20:27:30.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 14 tests from LibRadosIo (63983 ms total) 2023-10-30T20:27:30.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: 2023-10-30T20:27:30.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2023-10-30T20:27:30.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2023-10-30T20:27:30.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (6775 ms) 2023-10-30T20:27:30.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2023-10-30T20:27:30.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (471 ms) 2023-10-30T20:27:30.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2023-10-30T20:27:30.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (175 ms) 2023-10-30T20:27:30.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2023-10-30T20:27:30.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (278 ms) 2023-10-30T20:27:30.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2023-10-30T20:27:30.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (1376 ms) 2023-10-30T20:27:30.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2023-10-30T20:27:30.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (243 ms) 2023-10-30T20:27:30.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2023-10-30T20:27:30.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (154 ms) 2023-10-30T20:27:30.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2023-10-30T20:27:30.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (226 ms) 2023-10-30T20:27:30.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2023-10-30T20:27:30.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (787 ms) 2023-10-30T20:27:30.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2023-10-30T20:27:30.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (243 ms) 2023-10-30T20:27:30.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 10 tests from LibRadosIoEC (10729 ms total) 2023-10-30T20:27:30.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: 2023-10-30T20:27:30.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] Global test environment tear-down 2023-10-30T20:27:30.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [==========] 24 tests from 2 test suites ran. (133906 ms total) 2023-10-30T20:27:30.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ PASSED ] 24 tests. 2023-10-30T20:27:30.643 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:04:52.670 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:27:30.651 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:52.710 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:27:30.653 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:52.712 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:27:30.654 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: Running main() from gmock_main.cc 2023-10-30T20:27:30.655 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2023-10-30T20:27:30.655 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] Global test environment set-up. 2023-10-30T20:27:30.655 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 5 tests from LibRadosStat 2023-10-30T20:27:30.655 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2023-10-30T20:27:30.655 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.Stat (39062 ms) 2023-10-30T20:27:30.656 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2023-10-30T20:27:30.656 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (15610 ms) 2023-10-30T20:27:30.656 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2023-10-30T20:27:30.656 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.StatNS (6632 ms) 2023-10-30T20:27:30.656 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2023-10-30T20:27:30.656 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1656 ms) 2023-10-30T20:27:30.657 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2023-10-30T20:27:30.657 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (1947 ms) 2023-10-30T20:27:30.657 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 5 tests from LibRadosStat (64907 ms total) 2023-10-30T20:27:30.657 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: 2023-10-30T20:27:30.657 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2023-10-30T20:27:30.657 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2023-10-30T20:27:30.658 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (7361 ms) 2023-10-30T20:27:30.658 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2023-10-30T20:27:30.658 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (2577 ms) 2023-10-30T20:27:30.658 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2023-10-30T20:27:30.658 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (2240 ms) 2023-10-30T20:27:30.659 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2023-10-30T20:27:30.659 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1512 ms) 2023-10-30T20:27:30.659 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (13690 ms total) 2023-10-30T20:27:30.659 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: 2023-10-30T20:27:30.659 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] Global test environment tear-down 2023-10-30T20:27:30.659 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (133812 ms total) 2023-10-30T20:27:30.660 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ PASSED ] 9 tests. 2023-10-30T20:27:31.466 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: Running main() from gmock_main.cc 2023-10-30T20:27:31.466 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2023-10-30T20:27:31.467 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] Global test environment set-up. 2023-10-30T20:27:31.467 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2023-10-30T20:27:31.467 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2023-10-30T20:27:31.467 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify_test_cb 2023-10-30T20:27:31.467 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (47182 ms) 2023-10-30T20:27:31.468 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2023-10-30T20:27:31.468 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-10-30T20:27:31.468 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94009100102144 err -107 2023-10-30T20:27:31.468 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (13897 ms) 2023-10-30T20:27:31.468 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2023-10-30T20:27:31.468 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-10-30T20:27:31.469 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94009100102144 err -107 2023-10-30T20:27:31.469 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (2646 ms) 2023-10-30T20:27:31.469 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2023-10-30T20:27:31.469 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4397 notify_id 330712481792 cookie 94009102503312 2023-10-30T20:27:31.469 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (264 ms) 2023-10-30T20:27:31.470 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2023-10-30T20:27:31.470 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4397 notify_id 330712481792 cookie 94009102503312 2023-10-30T20:27:31.470 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (379 ms) 2023-10-30T20:27:31.470 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2023-10-30T20:27:31.470 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4397 notify_id 330712481792 cookie 94009101933024 2023-10-30T20:27:31.471 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (449 ms) 2023-10-30T20:27:31.471 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2023-10-30T20:27:31.471 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4397 notify_id 330712481792 cookie 94009101933024 2023-10-30T20:27:31.471 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4397 notify_id 330712481792 cookie 94009102591216 2023-10-30T20:27:31.471 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (3027 ms) 2023-10-30T20:27:31.471 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2023-10-30T20:27:31.472 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4397 notify_id 335007449089 cookie 94009101933024 2023-10-30T20:27:31.472 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4397 notify_id 335007449089 cookie 94009101933024 2023-10-30T20:27:31.472 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4397 notify_id 335007449090 cookie 94009101933024 2023-10-30T20:27:31.472 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4397 notify_id 335007449091 cookie 94009101933024 2023-10-30T20:27:31.472 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3680 ms) 2023-10-30T20:27:31.472 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2023-10-30T20:27:31.473 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2023-10-30T20:27:31.473 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94009101933024 err -107 2023-10-30T20:27:31.473 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4397 notify_id 347892350978 cookie 94009101933024 2023-10-30T20:27:31.473 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (6232 ms) 2023-10-30T20:27:31.473 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2023-10-30T20:27:31.473 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94009101933024 err -107 2023-10-30T20:27:31.474 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2023-10-30T20:27:31.474 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (155 ms) 2023-10-30T20:27:31.474 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (77911 ms total) 2023-10-30T20:27:31.474 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: 2023-10-30T20:27:31.474 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2023-10-30T20:27:31.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2023-10-30T20:27:31.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify_test_cb 2023-10-30T20:27:31.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (5148 ms) 2023-10-30T20:27:31.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (5148 ms total) 2023-10-30T20:27:31.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: 2023-10-30T20:27:31.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] Global test environment tear-down 2023-10-30T20:27:31.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (134610 ms total) 2023-10-30T20:27:31.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ PASSED ] 11 tests. 2023-10-30T20:27:39.560 INFO:tasks.workunit.client.0.smithi069.stdout:: listed object 0... 2023-10-30T20:27:39.561 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135447]: listed object 25... 2023-10-30T20:27:39.561 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135447]: listed object 50... 2023-10-30T20:27:39.561 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135447]: listed object 75... 2023-10-30T20:27:39.561 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135447]: listed object 100... 2023-10-30T20:27:39.562 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135447]: listed object 125... 2023-10-30T20:27:39.562 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135447]: saw 150 objects 2023-10-30T20:27:39.562 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135447]: shutting down. 2023-10-30T20:27:39.562 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:39.562 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:39.563 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:39.563 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:39.563 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[135448]: starting. 2023-10-30T20:27:39.563 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[135448]: added 25 objects... 2023-10-30T20:27:39.563 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[135448]: added half of the objects 2023-10-30T20:27:39.564 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[135448]: added 50 objects... 2023-10-30T20:27:39.564 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[135448]: added 50 objects 2023-10-30T20:27:39.564 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[135448]: shutting down. 2023-10-30T20:27:39.564 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:39.564 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:39.564 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:39.565 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:39.565 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-10-30T20:27:39.565 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******* SUCCESS ********** 2023-10-30T20:27:45.942 INFO:tasks.workunit.client.0.smithi069.stdout:dTripPP2 (298 ms) 2023-10-30T20:27:45.942 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2023-10-30T20:27:45.942 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (257 ms) 2023-10-30T20:27:45.942 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2023-10-30T20:27:45.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (1063 ms) 2023-10-30T20:27:45.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2023-10-30T20:27:45.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (295 ms) 2023-10-30T20:27:45.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2023-10-30T20:27:45.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (389 ms) 2023-10-30T20:27:45.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2023-10-30T20:27:45.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (133 ms) 2023-10-30T20:27:45.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2023-10-30T20:27:45.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (147 ms) 2023-10-30T20:27:45.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2023-10-30T20:27:45.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (143 ms) 2023-10-30T20:27:45.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2023-10-30T20:27:45.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (489 ms) 2023-10-30T20:27:45.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2023-10-30T20:27:45.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (285 ms) 2023-10-30T20:27:45.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2023-10-30T20:27:45.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (19844 ms) 2023-10-30T20:27:45.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2023-10-30T20:27:45.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (2148 ms) 2023-10-30T20:27:45.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2023-10-30T20:27:45.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (1399 ms) 2023-10-30T20:27:45.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (37570 ms total) 2023-10-30T20:27:45.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: 2023-10-30T20:27:45.947 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] Global test environment tear-down 2023-10-30T20:27:45.947 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (149261 ms total) 2023-10-30T20:27:45.947 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ PASSED ] 37 tests. 2023-10-30T20:28:10.650 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:32.708 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:10.715 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:32.773 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:10.718 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:32.745 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:10.724 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:32.751 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:10.749 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:32.807 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:10.794 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:32.852 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:10.846 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:32.873 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:10.859 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:32.918 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:10.926 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:32.985 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:13.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: Running main() from gmock_main.cc 2023-10-30T20:28:13.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2023-10-30T20:28:13.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] Global test environment set-up. 2023-10-30T20:28:13.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2023-10-30T20:28:13.972 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2023-10-30T20:28:13.972 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (17347 ms) 2023-10-30T20:28:13.972 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2023-10-30T20:28:13.972 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (15668 ms) 2023-10-30T20:28:13.972 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2023-10-30T20:28:13.973 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (13474 ms) 2023-10-30T20:28:13.973 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2023-10-30T20:28:13.973 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (4991 ms) 2023-10-30T20:28:13.973 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (51480 ms total) 2023-10-30T20:28:13.973 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2023-10-30T20:28:13.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2023-10-30T20:28:13.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2023-10-30T20:28:13.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (30126 ms) 2023-10-30T20:28:13.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2023-10-30T20:28:13.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (38451 ms) 2023-10-30T20:28:13.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2023-10-30T20:28:13.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (11433 ms) 2023-10-30T20:28:13.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (80010 ms total) 2023-10-30T20:28:13.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2023-10-30T20:28:13.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2023-10-30T20:28:13.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2023-10-30T20:28:13.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (5148 ms) 2023-10-30T20:28:13.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2023-10-30T20:28:13.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2185 ms) 2023-10-30T20:28:13.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2023-10-30T20:28:13.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (4612 ms) 2023-10-30T20:28:13.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2023-10-30T20:28:13.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2081 ms) 2023-10-30T20:28:13.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (14027 ms total) 2023-10-30T20:28:13.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2023-10-30T20:28:13.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2023-10-30T20:28:13.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2023-10-30T20:28:13.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (5811 ms) 2023-10-30T20:28:13.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2023-10-30T20:28:13.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (5874 ms) 2023-10-30T20:28:13.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (11685 ms total) 2023-10-30T20:28:13.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2023-10-30T20:28:13.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] Global test environment tear-down 2023-10-30T20:28:13.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (177180 ms total) 2023-10-30T20:28:13.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ PASSED ] 13 tests. 2023-10-30T20:28:30.888 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:52.946 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:30.905 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:52.932 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:30.959 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:53.018 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:30.969 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:52.996 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:30.990 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:53.049 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:30.994 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:53.053 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:31.022 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:53.080 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:31.094 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:53.121 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:31.096 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:53.155 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:33.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2023-10-30T20:28:33.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [==========] Running 19 tests from 4 test suites. 2023-10-30T20:28:33.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2023-10-30T20:28:33.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2023-10-30T20:28:33.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: seed 91906 2023-10-30T20:28:33.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2023-10-30T20:28:33.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (12462 ms) 2023-10-30T20:28:33.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2023-10-30T20:28:33.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (12150 ms) 2023-10-30T20:28:33.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2023-10-30T20:28:33.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (11846 ms) 2023-10-30T20:28:33.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2023-10-30T20:28:33.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (11181 ms) 2023-10-30T20:28:33.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2023-10-30T20:28:33.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (27407 ms) 2023-10-30T20:28:33.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (75046 ms total) 2023-10-30T20:28:33.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-10-30T20:28:33.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2023-10-30T20:28:33.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2023-10-30T20:28:33.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (25968 ms) 2023-10-30T20:28:33.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2023-10-30T20:28:33.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (8687 ms) 2023-10-30T20:28:33.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2023-10-30T20:28:33.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (8661 ms) 2023-10-30T20:28:33.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2023-10-30T20:28:33.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4606 ms) 2023-10-30T20:28:33.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2023-10-30T20:28:33.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (3612 ms) 2023-10-30T20:28:33.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-10-30T20:28:33.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6839-gcc36dd85/rpm/el9/BUILD/ceph-18.0.0-6839-gcc36dd85/src/test/librados/snapshots_cxx.cc:463: Skipped 2023-10-30T20:28:33.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-10-30T20:28:33.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2023-10-30T20:28:33.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2023-10-30T20:28:33.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi069-91906-7 2023-10-30T20:28:33.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: waiting for snaps to purge 2023-10-30T20:28:33.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17406 ms) 2023-10-30T20:28:33.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (68940 ms total) 2023-10-30T20:28:33.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-10-30T20:28:33.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2023-10-30T20:28:33.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2023-10-30T20:28:33.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2564 ms) 2023-10-30T20:28:33.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2023-10-30T20:28:33.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2176 ms) 2023-10-30T20:28:33.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2023-10-30T20:28:33.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2144 ms) 2023-10-30T20:28:33.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2023-10-30T20:28:33.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2988 ms) 2023-10-30T20:28:33.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9873 ms total) 2023-10-30T20:28:33.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-10-30T20:28:33.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2023-10-30T20:28:46.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN api_aio: Running main() from gmock_main.cc 2023-10-30T20:28:46.921 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2023-10-30T20:28:46.921 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] Global test environment set-up. 2023-10-30T20:28:46.921 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 26 tests from LibRadosAio 2023-10-30T20:28:46.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2023-10-30T20:28:46.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.TooBig (4467 ms) 2023-10-30T20:28:46.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2023-10-30T20:28:46.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (36336 ms) 2023-10-30T20:28:46.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2023-10-30T20:28:46.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (10158 ms) 2023-10-30T20:28:46.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2023-10-30T20:28:46.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (7142 ms) 2023-10-30T20:28:46.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2023-10-30T20:28:46.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (11075 ms) 2023-10-30T20:28:46.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2023-10-30T20:28:46.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (27669 ms) 2023-10-30T20:28:46.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2023-10-30T20:28:46.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (30801 ms) 2023-10-30T20:28:46.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2023-10-30T20:28:46.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (13052 ms) 2023-10-30T20:28:46.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2023-10-30T20:28:46.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (4273 ms) 2023-10-30T20:28:46.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2023-10-30T20:28:46.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (4305 ms) 2023-10-30T20:28:46.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2023-10-30T20:28:46.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3109 ms) 2023-10-30T20:28:46.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2023-10-30T20:28:46.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3573 ms) 2023-10-30T20:28:46.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2023-10-30T20:28:46.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3846 ms) 2023-10-30T20:28:46.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2023-10-30T20:28:46.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3116 ms) 2023-10-30T20:28:46.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2023-10-30T20:28:46.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.Flush (4056 ms) 2023-10-30T20:28:46.928 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2023-10-30T20:28:46.928 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (4378 ms) 2023-10-30T20:28:46.928 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2023-10-30T20:28:46.928 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (4866 ms) 2023-10-30T20:28:46.928 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2023-10-30T20:28:46.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (4168 ms) 2023-10-30T20:28:46.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2023-10-30T20:28:46.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3643 ms) 2023-10-30T20:28:46.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2023-10-30T20:28:46.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (4327 ms) 2023-10-30T20:28:46.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2023-10-30T20:28:46.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (2983 ms) 2023-10-30T20:28:46.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2023-10-30T20:28:46.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3883 ms) 2023-10-30T20:28:46.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2023-10-30T20:28:46.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (4085 ms) 2023-10-30T20:28:46.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2023-10-30T20:28:46.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (4090 ms) 2023-10-30T20:28:46.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2023-10-30T20:28:46.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3923 ms) 2023-10-30T20:28:46.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2023-10-30T20:28:46.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (2931 ms) 2023-10-30T20:28:46.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 26 tests from LibRadosAio (210256 ms total) 2023-10-30T20:28:46.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: 2023-10-30T20:28:46.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2023-10-30T20:28:48.003 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.Simple api_aio_pp: Running main() from gmock_main.cc 2023-10-30T20:28:48.004 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2023-10-30T20:28:48.004 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] Global test environment set-up. 2023-10-30T20:28:48.004 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2023-10-30T20:28:48.004 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2023-10-30T20:28:48.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (4435 ms) 2023-10-30T20:28:48.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2023-10-30T20:28:48.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (36463 ms) 2023-10-30T20:28:48.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2023-10-30T20:28:48.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (15074 ms) 2023-10-30T20:28:48.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2023-10-30T20:28:48.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (13467 ms) 2023-10-30T20:28:48.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2023-10-30T20:28:48.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (27403 ms) 2023-10-30T20:28:48.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2023-10-30T20:28:48.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (23484 ms) 2023-10-30T20:28:48.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2023-10-30T20:28:48.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (16502 ms) 2023-10-30T20:28:48.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2023-10-30T20:28:48.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2768 ms) 2023-10-30T20:28:48.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2023-10-30T20:28:48.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3316 ms) 2023-10-30T20:28:48.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2023-10-30T20:28:48.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (4109 ms) 2023-10-30T20:28:48.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2023-10-30T20:28:48.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3223 ms) 2023-10-30T20:28:48.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2023-10-30T20:28:48.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (4596 ms) 2023-10-30T20:28:48.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2023-10-30T20:28:48.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3179 ms) 2023-10-30T20:28:48.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2023-10-30T20:28:48.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3835 ms) 2023-10-30T20:28:48.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2023-10-30T20:28:48.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3022 ms) 2023-10-30T20:28:48.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2023-10-30T20:28:48.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (4031 ms) 2023-10-30T20:28:48.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2023-10-30T20:28:48.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (4599 ms) 2023-10-30T20:28:48.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2023-10-30T20:28:48.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2682 ms) 2023-10-30T20:28:48.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2023-10-30T20:28:48.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3176 ms) 2023-10-30T20:28:48.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2023-10-30T20:28:48.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3087 ms) 2023-10-30T20:28:48.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2023-10-30T20:28:48.013 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2637 ms) 2023-10-30T20:28:48.013 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2023-10-30T20:28:48.013 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (4004 ms) 2023-10-30T20:28:48.013 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2023-10-30T20:28:48.013 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (4336 ms) 2023-10-30T20:28:48.013 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2023-10-30T20:28:48.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (5918 ms) 2023-10-30T20:28:48.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2023-10-30T20:28:48.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (4058 ms) 2023-10-30T20:28:48.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2023-10-30T20:28:48.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3913 ms) 2023-10-30T20:28:48.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2023-10-30T20:28:50.975 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:13.002 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:50.978 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:13.005 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:50.984 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:13.042 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:50.987 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:13.046 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:51.042 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:13.069 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:51.044 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:13.103 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:51.050 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:13.108 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:51.051 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:13.110 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:51.118 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:13.177 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:28:54.056 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRad] LibRadosSnapshotsSelfManagedECPP.SnapPP 2023-10-30T20:28:54.057 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (6869 ms) 2023-10-30T20:28:54.057 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2023-10-30T20:28:54.057 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (6091 ms) 2023-10-30T20:28:54.057 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2023-10-30T20:28:54.057 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3974 ms) 2023-10-30T20:28:54.057 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (16934 ms total) 2023-10-30T20:28:54.058 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-10-30T20:28:54.058 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2023-10-30T20:28:54.058 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [==========] 19 tests from 4 test suites ran. (217238 ms total) 2023-10-30T20:28:54.058 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2023-10-30T20:28:54.058 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2023-10-30T20:28:54.059 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-10-30T20:29:11.034 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:33.092 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:11.088 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:33.146 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:11.100 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:33.127 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:11.103 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:33.130 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:11.145 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:33.203 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:11.152 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:33.210 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:11.153 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:33.212 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:11.160 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:33.187 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:11.212 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:33.271 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:31.037 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:53.095 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:31.041 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:53.100 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:31.110 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:53.168 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:31.111 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:53.137 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:31.113 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:53.172 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:31.124 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:53.151 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:31.130 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:53.189 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:31.191 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:53.218 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:31.218 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:53.276 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:32.848 INFO:tasks.workunit.client.0.smithi069.stdout:: cursor()=8:e960b815:::9:head expected=8:e960b815:::9:head 2023-10-30T20:29:32.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:e960b815:::9:head -> 9 2023-10-30T20:29:32.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=9 expected=9 2023-10-30T20:29:32.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:d83876eb:::4:head 2023-10-30T20:29:32.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:d83876eb:::4:head expected=8:d83876eb:::4:head 2023-10-30T20:29:32.850 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:d83876eb:::4:head -> 4 2023-10-30T20:29:32.850 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=4 expected=4 2023-10-30T20:29:32.850 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:89d3ae78:::11:head 2023-10-30T20:29:32.850 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:89d3ae78:::11:head expected=8:89d3ae78:::11:head 2023-10-30T20:29:32.850 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2023-10-30T20:29:32.851 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=11 expected=11 2023-10-30T20:29:32.851 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:863748b0:::15:head 2023-10-30T20:29:32.851 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:863748b0:::15:head expected=8:863748b0:::15:head 2023-10-30T20:29:32.851 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:863748b0:::15:head -> 15 2023-10-30T20:29:32.851 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=15 expected=15 2023-10-30T20:29:32.852 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:6cac518f:::0:head 2023-10-30T20:29:32.852 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:6cac518f:::0:head expected=8:6cac518f:::0:head 2023-10-30T20:29:32.852 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:6cac518f:::0:head -> 0 2023-10-30T20:29:32.852 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=0 expected=0 2023-10-30T20:29:32.852 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:62a1935d:::14:head 2023-10-30T20:29:32.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:62a1935d:::14:head expected=8:62a1935d:::14:head 2023-10-30T20:29:32.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:62a1935d:::14:head -> 14 2023-10-30T20:29:32.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=14 expected=14 2023-10-30T20:29:32.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:566253c9:::13:head 2023-10-30T20:29:32.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:566253c9:::13:head expected=8:566253c9:::13:head 2023-10-30T20:29:32.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:566253c9:::13:head -> 13 2023-10-30T20:29:32.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=13 expected=13 2023-10-30T20:29:32.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2023-10-30T20:29:32.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:de5d7c5f:::12:head expected=8:de5d7c5f:::12:head 2023-10-30T20:29:32.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2023-10-30T20:29:32.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=12 expected=12 2023-10-30T20:29:32.855 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (11069 ms) 2023-10-30T20:29:32.855 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2023-10-30T20:29:32.855 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (132138 ms) 2023-10-30T20:29:32.855 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2023-10-30T20:29:32.855 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 0/5 -> MIN 8:33333333::::head 2023-10-30T20:29:32.856 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 1/5 -> 8:33333333::::head 8:66666666::::head 2023-10-30T20:29:32.856 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 2/5 -> 8:66666666::::head 8:99999999::::head 2023-10-30T20:29:32.856 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 3/5 -> 8:99999999::::head 8:cccccccc::::head 2023-10-30T20:29:32.856 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 4/5 -> 8:cccccccc::::head MAX 2023-10-30T20:29:32.856 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (15762 ms) 2023-10-30T20:29:32.857 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 7 tests from LibRadosList (239801 ms total) 2023-10-30T20:29:32.857 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2023-10-30T20:29:32.857 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 3 tests from LibRadosListEC 2023-10-30T20:29:32.857 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2023-10-30T20:29:32.857 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (3332 ms) 2023-10-30T20:29:32.858 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2023-10-30T20:29:32.858 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo2,foo3 2023-10-30T20:29:32.858 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2023-10-30T20:29:32.858 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo2 2023-10-30T20:29:32.858 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo3 2023-10-30T20:29:32.859 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo4,foo5 2023-10-30T20:29:32.859 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo4 2023-10-30T20:29:32.859 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo5 2023-10-30T20:29:32.859 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2023-10-30T20:29:32.859 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo6,foo7 2023-10-30T20:29:32.859 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo7 2023-10-30T20:29:32.860 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo6 2023-10-30T20:29:32.860 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-10-30T20:29:32.860 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo4 2023-10-30T20:29:32.860 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo5 2023-10-30T20:29:32.860 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo7 2023-10-30T20:29:32.860 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo6 2023-10-30T20:29:32.861 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo1 2023-10-30T20:29:32.861 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo1 2023-10-30T20:29:32.861 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo2 2023-10-30T20:29:32.861 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo3 2023-10-30T20:29:32.861 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (3243 ms) 2023-10-30T20:29:32.862 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2023-10-30T20:29:32.862 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 1 0 2023-10-30T20:29:32.862 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 10 0 2023-10-30T20:29:32.862 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 13 0 2023-10-30T20:29:41.140 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 7 0 2023-10-30T20:29:41.141 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 14 0 2023-10-30T20:29:41.141 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 0 0 2023-10-30T20:29:41.141 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 15 0 2023-10-30T20:29:41.141 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 11 0 2023-10-30T20:29:41.142 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 5 0 2023-10-30T20:29:41.142 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 8 0 2023-10-30T20:29:41.142 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 6 0 2023-10-30T20:29:41.142 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 3 0 2023-10-30T20:29:41.142 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 4 0 2023-10-30T20:29:41.143 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 12 0 2023-10-30T20:29:41.143 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 9 0 2023-10-30T20:29:41.143 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2 0 2023-10-30T20:29:41.143 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-10-30T20:29:41.143 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (4626 ms) 2023-10-30T20:29:41.144 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 3 tests from LibRadosListEC (11201 ms total) 2023-10-30T20:29:41.144 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2023-10-30T20:29:41.144 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 1 test from LibRadosListNP 2023-10-30T20:29:41.144 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2023-10-30T20:29:41.144 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3595 ms) 2023-10-30T20:29:41.144 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 1 test from LibRadosListNP (3595 ms total) 2023-10-30T20:29:41.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2023-10-30T20:29:41.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] Global test environment tear-down 2023-10-30T20:29:41.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [==========] 11 tests from 3 test suites ran. (264427 ms total) 2023-10-30T20:29:41.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ PASSED ] 11 tests. 2023-10-30T20:29:43.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: Running main() from gmock_main.cc 2023-10-30T20:29:43.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2023-10-30T20:29:43.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] Global test environment set-up. 2023-10-30T20:29:43.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2023-10-30T20:29:43.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2023-10-30T20:29:43.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2023-10-30T20:29:43.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2023-10-30T20:29:43.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: 2023-10-30T20:29:43.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2023-10-30T20:29:43.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2023-10-30T20:29:43.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (22982 ms) 2023-10-30T20:29:43.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2023-10-30T20:29:43.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (6668 ms) 2023-10-30T20:29:43.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2023-10-30T20:29:43.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (29273 ms) 2023-10-30T20:29:43.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2023-10-30T20:29:43.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (12551 ms) 2023-10-30T20:29:43.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2023-10-30T20:29:43.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (23764 ms) 2023-10-30T20:29:43.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2023-10-30T20:29:43.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (28355 ms) 2023-10-30T20:29:43.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2023-10-30T20:29:43.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (142638 ms) 2023-10-30T20:29:43.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (266231 ms total) 2023-10-30T20:29:43.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: 2023-10-30T20:29:43.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] Global test environment tear-down 2023-10-30T20:29:43.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (266231 ms total) 2023-10-30T20:29:43.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2023-10-30T20:29:51.156 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:13.214 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:51.205 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:13.232 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:51.207 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:13.234 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:51.217 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:13.275 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:51.220 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:13.279 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:51.229 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:13.288 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:51.266 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:13.293 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:51.282 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:13.340 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:51.296 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:13.355 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:29:57.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2023-10-30T20:29:57.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] Global test environment set-up. 2023-10-30T20:29:57.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2023-10-30T20:29:57.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2023-10-30T20:29:57.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2023-10-30T20:29:57.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-10-30T20:29:57.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-10-30T20:29:57.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2023-10-30T20:29:57.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: seed 91719 2023-10-30T20:29:57.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2023-10-30T20:29:57.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (7 ms) 2023-10-30T20:29:57.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2023-10-30T20:29:57.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (26233 ms) 2023-10-30T20:29:57.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2023-10-30T20:29:57.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (18643 ms) 2023-10-30T20:29:57.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2023-10-30T20:29:57.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (8081 ms) 2023-10-30T20:29:57.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2023-10-30T20:29:57.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (3467 ms) 2023-10-30T20:29:57.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2023-10-30T20:29:57.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (3114 ms) 2023-10-30T20:29:57.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2023-10-30T20:29:57.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (1280 ms) 2023-10-30T20:29:57.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2023-10-30T20:29:57.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (612 ms) 2023-10-30T20:29:57.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2023-10-30T20:29:57.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (784 ms) 2023-10-30T20:29:57.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2023-10-30T20:29:57.600 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (1065 ms) 2023-10-30T20:29:57.600 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2023-10-30T20:29:57.600 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (159 ms) 2023-10-30T20:29:57.600 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2023-10-30T20:29:57.600 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (219 ms) 2023-10-30T20:29:57.600 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2023-10-30T20:29:57.601 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (262 ms) 2023-10-30T20:29:57.601 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2023-10-30T20:29:57.601 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: osd_max_attr_size = 0 2023-10-30T20:29:57.601 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2023-10-30T20:29:57.601 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (116877 ms) 2023-10-30T20:29:57.601 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2023-10-30T20:29:57.602 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (11179 ms) 2023-10-30T20:29:57.602 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2023-10-30T20:29:57.602 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: waiting for initial deep scrubs... 2023-10-30T20:29:57.602 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: done waiting, doing copies 2023-10-30T20:29:57.602 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: waiting for final deep scrubs... 2023-10-30T20:29:57.603 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: done waiting 2023-10-30T20:29:57.603 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (78968 ms) 2023-10-30T20:29:57.603 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2023-10-30T20:29:57.603 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (494 ms) 2023-10-30T20:29:57.603 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2023-10-30T20:29:57.603 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (63 ms) 2023-10-30T20:29:57.604 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2023-10-30T20:29:57.604 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5192 ms) 2023-10-30T20:29:57.604 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2023-10-30T20:29:57.604 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2023-10-30T20:29:57.604 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2023-10-30T20:29:57.604 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2023-10-30T20:29:57.605 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2023-10-30T20:29:57.605 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (1 ms) 2023-10-30T20:30:11.339 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:33.397 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:11.342 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:33.400 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:11.400 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:33.427 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:11.402 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:33.429 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:11.417 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:33.475 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:11.418 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:33.477 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:11.419 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:33.478 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:11.457 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:33.484 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:11.481 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:33.539 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:13.575 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:35.601 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:13.577 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:35.604 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:13.579 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:35.638 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:13.583 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:35.642 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:13.610 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:35.637 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:13.631 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:35.689 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:13.666 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:35.725 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:13.671 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:35.730 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:13.693 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:35.752 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:25.862 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP (276700 ms total) 2023-10-30T20:30:25.862 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-10-30T20:30:25.863 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2023-10-30T20:30:25.863 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2023-10-30T20:30:25.863 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (2853 ms) 2023-10-30T20:30:25.863 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (2853 ms total) 2023-10-30T20:30:25.863 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-10-30T20:30:25.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2023-10-30T20:30:25.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2023-10-30T20:30:25.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (489 ms) 2023-10-30T20:30:25.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2023-10-30T20:30:25.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (36 ms) 2023-10-30T20:30:25.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (525 ms total) 2023-10-30T20:30:25.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-10-30T20:30:25.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2023-10-30T20:30:25.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2023-10-30T20:30:25.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (355 ms) 2023-10-30T20:30:25.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2023-10-30T20:30:25.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (65 ms) 2023-10-30T20:30:25.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (420 ms total) 2023-10-30T20:30:25.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-10-30T20:30:25.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2023-10-30T20:30:25.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2023-10-30T20:30:25.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (326 ms) 2023-10-30T20:30:25.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2023-10-30T20:30:25.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (44 ms) 2023-10-30T20:30:25.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (371 ms total) 2023-10-30T20:30:25.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-10-30T20:30:25.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2023-10-30T20:30:25.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2023-10-30T20:30:25.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1635 ms) 2023-10-30T20:30:25.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1635 ms total) 2023-10-30T20:30:25.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-10-30T20:30:25.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] Global test environment tear-down 2023-10-30T20:30:25.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (309108 ms total) 2023-10-30T20:30:25.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ PASSED ] 31 tests. 2023-10-30T20:30:31.518 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:53.576 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:31.522 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:53.579 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:31.566 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:53.624 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:31.573 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:53.600 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:31.574 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:53.601 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:31.586 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:53.644 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:31.589 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:53.648 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:31.616 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:53.643 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:31.639 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:53.698 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:46.560 INFO:tasks.workunit.client.0.smithi069.stdout:Write 2023-10-30T20:30:46.561 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (8216 ms) 2023-10-30T20:30:46.561 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2023-10-30T20:30:46.561 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6789 ms) 2023-10-30T20:30:46.561 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2023-10-30T20:30:46.561 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7905 ms) 2023-10-30T20:30:46.562 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2023-10-30T20:30:46.562 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7224 ms) 2023-10-30T20:30:46.562 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2023-10-30T20:30:46.562 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (8894 ms) 2023-10-30T20:30:46.562 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2023-10-30T20:30:46.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (8171 ms) 2023-10-30T20:30:46.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2023-10-30T20:30:46.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7026 ms) 2023-10-30T20:30:46.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2023-10-30T20:30:46.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7304 ms) 2023-10-30T20:30:46.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2023-10-30T20:30:46.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7095 ms) 2023-10-30T20:30:46.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2023-10-30T20:30:46.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6737 ms) 2023-10-30T20:30:46.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2023-10-30T20:30:46.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7554 ms) 2023-10-30T20:30:46.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2023-10-30T20:30:46.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7067 ms) 2023-10-30T20:30:46.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2023-10-30T20:30:46.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7861 ms) 2023-10-30T20:30:46.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2023-10-30T20:30:46.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7696 ms) 2023-10-30T20:30:46.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2023-10-30T20:30:46.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7248 ms) 2023-10-30T20:30:46.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2023-10-30T20:30:46.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6850 ms) 2023-10-30T20:30:46.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (119637 ms total) 2023-10-30T20:30:46.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: 2023-10-30T20:30:46.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] Global test environment tear-down 2023-10-30T20:30:46.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (329893 ms total) 2023-10-30T20:30:46.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ PASSED ] 42 tests. 2023-10-30T20:30:51.582 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:13.640 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:51.586 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:13.644 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:51.589 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:13.646 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:51.616 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:13.675 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:51.643 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:13.670 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:51.647 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:13.674 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:51.653 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:13.711 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:51.692 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:13.719 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:30:51.728 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:13.786 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:11.624 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:33.683 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:11.627 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:33.686 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:11.663 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:33.721 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:11.671 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:33.698 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:11.672 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:33.731 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:11.673 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:33.700 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:11.676 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:33.734 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:11.707 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:33.733 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:11.712 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:33.770 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:24.205 INFO:tasks.workunit.client.0.smithi069.stdout:osAio.RoundTripAppendPP (4019 ms) 2023-10-30T20:31:24.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2023-10-30T20:31:24.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2890 ms) 2023-10-30T20:31:24.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2023-10-30T20:31:24.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3132 ms) 2023-10-30T20:31:24.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2023-10-30T20:31:24.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3177 ms) 2023-10-30T20:31:24.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2023-10-30T20:31:24.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: setting pool EIO 2023-10-30T20:31:24.207 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: max_success 34, min_failed 35 2023-10-30T20:31:24.207 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (7005 ms) 2023-10-30T20:31:24.207 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2023-10-30T20:31:24.207 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4574 ms) 2023-10-30T20:31:24.207 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (232114 ms total) 2023-10-30T20:31:24.207 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2023-10-30T20:31:24.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2023-10-30T20:31:24.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2023-10-30T20:31:24.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3342 ms) 2023-10-30T20:31:24.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2023-10-30T20:31:24.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9346 ms) 2023-10-30T20:31:24.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2023-10-30T20:31:24.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (18302 ms) 2023-10-30T20:31:24.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2023-10-30T20:31:24.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (2390 ms) 2023-10-30T20:31:24.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (33380 ms total) 2023-10-30T20:31:24.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2023-10-30T20:31:24.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2023-10-30T20:31:24.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-10-30T20:31:24.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3039 ms) 2023-10-30T20:31:24.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3039 ms total) 2023-10-30T20:31:24.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2023-10-30T20:31:24.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2023-10-30T20:31:24.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2023-10-30T20:31:24.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14406 ms) 2023-10-30T20:31:24.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2023-10-30T20:31:24.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6848 ms) 2023-10-30T20:31:24.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2023-10-30T20:31:24.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (8281 ms) 2023-10-30T20:31:24.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2023-10-30T20:31:24.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6760 ms) 2023-10-30T20:31:24.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2023-10-30T20:31:24.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3261 ms) 2023-10-30T20:31:24.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2023-10-30T20:31:24.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7672 ms) 2023-10-30T20:31:24.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2023-10-30T20:31:24.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7287 ms) 2023-10-30T20:31:24.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2023-10-30T20:31:24.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7807 ms) 2023-10-30T20:31:24.214 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2023-10-30T20:31:24.214 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7278 ms) 2023-10-30T20:31:24.214 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2023-10-30T20:31:24.214 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7053 ms) 2023-10-30T20:31:24.214 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2023-10-30T20:31:24.215 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7953 ms) 2023-10-30T20:31:24.215 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2023-10-30T20:31:24.215 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6834 ms) 2023-10-30T20:31:24.215 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2023-10-30T20:31:24.215 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7564 ms) 2023-10-30T20:31:31.637 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:53.695 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:31.640 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:53.699 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:31.678 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:53.736 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:31.683 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:53.710 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:31.684 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:53.743 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:31.686 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:53.713 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:31.689 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:53.748 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:31.722 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:53.749 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:31.728 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:53.787 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:51.788 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:13.846 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:51.847 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:13.905 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:51.856 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:13.883 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:51.860 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:13.887 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:51.904 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:13.962 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:51.907 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:13.934 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:51.921 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:13.980 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:51.924 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:13.983 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:31:51.980 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:14.038 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:01.259 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:23.317 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:01.312 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:23.370 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:01.348 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:23.406 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:01.380 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:23.407 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:01.390 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:23.417 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:01.414 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:23.472 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:01.425 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:23.452 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:01.426 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:23.485 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:01.460 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:23.519 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:11.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2023-10-30T20:32:11.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3473 ms) 2023-10-30T20:32:11.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2023-10-30T20:32:11.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7883 ms) 2023-10-30T20:32:11.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2023-10-30T20:32:11.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7377 ms) 2023-10-30T20:32:11.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2023-10-30T20:32:11.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7009 ms) 2023-10-30T20:32:11.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2023-10-30T20:32:11.251 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6986 ms) 2023-10-30T20:32:11.251 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2023-10-30T20:32:11.251 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7078 ms) 2023-10-30T20:32:11.251 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2023-10-30T20:32:11.251 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7234 ms) 2023-10-30T20:32:11.252 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (146044 ms total) 2023-10-30T20:32:11.252 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2023-10-30T20:32:11.252 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] Global test environment tear-down 2023-10-30T20:32:11.252 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (414577 ms total) 2023-10-30T20:32:11.252 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ PASSED ] 57 tests. 2023-10-30T20:32:11.878 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:33.936 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:11.880 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:33.938 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:11.907 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:33.965 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:11.936 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:33.963 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:11.939 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:33.966 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:11.942 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:34.000 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:11.943 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:34.002 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:11.965 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:33.992 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:11.974 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:34.033 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:31.850 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:53.908 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:31.854 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:53.912 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:31.880 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:53.938 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:31.899 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:53.926 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:31.902 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:53.929 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:31.902 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:53.961 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:31.907 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:53.966 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:31.920 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:53.946 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:31.925 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:53.984 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:51.932 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:13.990 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:51.937 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:13.995 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:51.975 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:14.034 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:51.986 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:14.013 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:51.988 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:14.015 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:51.988 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:14.047 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:51.991 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:14.050 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:52.019 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:14.046 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:32:52.023 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:14.082 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:12.072 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:34.130 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:12.073 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:34.132 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:12.104 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:34.163 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:12.117 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:34.144 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:12.118 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:34.145 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:12.120 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:34.179 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:12.121 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:34.180 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:12.149 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:34.176 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:12.153 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:34.212 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:31.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2023-10-30T20:33:31.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] Global test environment set-up. 2023-10-30T20:33:31.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2023-10-30T20:33:31.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: seed 91800 2023-10-30T20:33:31.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2023-10-30T20:33:31.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (53530 ms) 2023-10-30T20:33:31.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2023-10-30T20:33:31.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (61222 ms) 2023-10-30T20:33:31.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2023-10-30T20:33:31.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (280 ms) 2023-10-30T20:33:31.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (115032 ms total) 2023-10-30T20:33:31.079 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2023-10-30T20:33:31.079 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2023-10-30T20:33:31.079 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2023-10-30T20:33:31.079 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (17172 ms) 2023-10-30T20:33:31.080 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2023-10-30T20:33:31.080 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (16710 ms) 2023-10-30T20:33:31.080 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2023-10-30T20:33:31.081 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (21409 ms) 2023-10-30T20:33:31.081 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2023-10-30T20:33:31.081 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [4] 2023-10-30T20:33:31.081 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [5,4] 2023-10-30T20:33:31.082 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [6,5,4] 2023-10-30T20:33:31.082 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [7,6,5,4] 2023-10-30T20:33:31.082 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting some heads 2023-10-30T20:33:31.082 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 7 2023-10-30T20:33:31.083 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 6 2023-10-30T20:33:31.083 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 5 2023-10-30T20:33:31.083 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 4 2023-10-30T20:33:31.083 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrubs... 2023-10-30T20:33:31.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2023-10-30T20:33:31.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (143610 ms) 2023-10-30T20:33:31.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2023-10-30T20:33:31.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (11056 ms) 2023-10-30T20:33:31.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2023-10-30T20:33:31.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (10063 ms) 2023-10-30T20:33:31.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2023-10-30T20:33:31.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8887 ms) 2023-10-30T20:33:31.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2023-10-30T20:33:31.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (11185 ms) 2023-10-30T20:33:31.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2023-10-30T20:33:31.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (11219 ms) 2023-10-30T20:33:31.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2023-10-30T20:33:31.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8898 ms) 2023-10-30T20:33:31.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2023-10-30T20:33:31.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10004 ms) 2023-10-30T20:33:31.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2023-10-30T20:33:31.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (34325 ms) 2023-10-30T20:33:31.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2023-10-30T20:33:31.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8976 ms) 2023-10-30T20:33:31.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2023-10-30T20:33:31.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (10985 ms) 2023-10-30T20:33:31.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2023-10-30T20:33:31.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (14852 ms) 2023-10-30T20:33:31.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2023-10-30T20:33:31.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (9070 ms) 2023-10-30T20:33:31.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2023-10-30T20:33:31.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (12920 ms) 2023-10-30T20:33:32.182 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:54.240 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:32.190 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:54.248 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:32.190 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:54.217 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:32.191 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:54.218 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:32.239 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:54.297 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:32.246 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:54.305 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:32.288 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:54.315 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:32.290 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:54.349 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:32.332 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:54.390 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:52.230 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:14.288 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:52.233 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:14.291 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:52.265 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:14.323 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:52.279 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:14.306 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:52.281 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:14.308 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:52.285 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:14.343 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:52.290 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:14.349 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:52.307 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:14.334 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:52.320 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:14.378 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:33:53.338 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2023-10-30T20:33:53.339 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: hmm, no HitSet yet 2023-10-30T20:33:53.339 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: ok, hit_set contains 200:602f83fe:::foo:head 2023-10-30T20:33:53.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (10174 ms) 2023-10-30T20:33:53.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2023-10-30T20:33:53.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg_num = 32 2023-10-30T20:33:53.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 0 ls 1698698028,0 2023-10-30T20:33:53.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 1 ls 1698698028,0 2023-10-30T20:33:53.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 2 ls 1698698028,0 2023-10-30T20:33:53.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 3 ls 1698698028,0 2023-10-30T20:33:53.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 4 ls 1698698028,0 2023-10-30T20:33:53.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 5 ls 1698698028,0 2023-10-30T20:33:53.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 6 ls 1698698028,0 2023-10-30T20:33:53.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 7 ls 1698698028,0 2023-10-30T20:33:53.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 8 ls 1698698028,0 2023-10-30T20:33:53.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 9 ls 1698698028,0 2023-10-30T20:33:53.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 10 ls 1698698028,0 2023-10-30T20:33:53.344 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 11 ls 1698698028,0 2023-10-30T20:33:53.344 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 12 ls 1698698028,0 2023-10-30T20:33:53.344 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 13 ls 1698698028,0 2023-10-30T20:33:53.344 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 14 ls 1698698028,0 2023-10-30T20:33:53.345 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 15 ls 1698698028,0 2023-10-30T20:33:53.345 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 16 ls 1698698028,0 2023-10-30T20:33:53.345 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 17 ls 1698698028,0 2023-10-30T20:33:53.345 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 18 ls 1698698028,0 2023-10-30T20:33:53.346 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 19 ls 1698698028,0 2023-10-30T20:33:53.346 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 20 ls 1698698028,0 2023-10-30T20:33:53.346 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 21 ls 1698698028,0 2023-10-30T20:33:53.346 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 22 ls 1698698028,0 2023-10-30T20:33:53.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 23 ls 1698698028,0 2023-10-30T20:33:53.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 24 ls 1698698028,0 2023-10-30T20:33:53.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 25 ls 1698698028,0 2023-10-30T20:33:53.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 26 ls 1698698028,0 2023-10-30T20:33:53.348 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 27 ls 1698698028,0 2023-10-30T20:33:53.348 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 28 ls 1698698028,0 2023-10-30T20:33:53.348 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 29 ls 1698698028,0 2023-10-30T20:33:53.348 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 30 ls 1698698028,0 2023-10-30T20:33:53.349 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 31 ls 1698698028,0 2023-10-30T20:33:53.349 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg_num = 32 2023-10-30T20:33:53.349 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6cac518f:::0:head 2023-10-30T20:33:53.350 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:02547ec2:::1:head 2023-10-30T20:33:53.350 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f905c69b:::2:head 2023-10-30T20:33:53.350 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:cfc208b3:::3:head 2023-10-30T20:33:53.350 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d83876eb:::4:head 2023-10-30T20:33:53.351 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b29083e3:::5:head 2023-10-30T20:33:53.351 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c4fdafeb:::6:head 2023-10-30T20:33:53.351 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:5c6b0b28:::7:head 2023-10-30T20:33:53.351 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:bd63b0f1:::8:head 2023-10-30T20:33:53.352 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:e960b815:::9:head 2023-10-30T20:33:53.352 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:52ea6a34:::10:head 2023-10-30T20:33:53.352 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:89d3ae78:::11:head 2023-10-30T20:33:53.352 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:de5d7c5f:::12:head 2023-10-30T20:33:53.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:566253c9:::13:head 2023-10-30T20:33:53.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:62a1935d:::14:head 2023-10-30T20:33:53.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:863748b0:::15:head 2023-10-30T20:33:53.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:3958e169:::16:head 2023-10-30T20:33:53.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:4d4dabf9:::17:head 2023-10-30T20:33:53.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:8391935d:::18:head 2023-10-30T20:33:53.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:28883081:::19:head 2023-10-30T20:33:53.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:69259c59:::20:head 2023-10-30T20:33:53.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:4bdb80b7:::21:head 2023-10-30T20:33:53.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a11c5d71:::22:head 2023-10-30T20:33:53.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:271af37b:::23:head 2023-10-30T20:33:53.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:95b121be:::24:head 2023-10-30T20:33:53.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:58d1031b:::25:head 2023-10-30T20:33:53.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:0a050783:::26:head 2023-10-30T20:33:53.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c709704c:::27:head 2023-10-30T20:33:53.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:cbe56eaf:::28:head 2023-10-30T20:33:53.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:86b4b162:::29:head 2023-10-30T20:33:53.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:70d89383:::30:head 2023-10-30T20:33:53.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:dd450c7c:::31:head 2023-10-30T20:33:53.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6d5729b1:::32:head 2023-10-30T20:33:53.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c388f3fb:::33:head 2023-10-30T20:33:53.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:56cfea31:::34:head 2023-10-30T20:33:53.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:9dbc1bf7:::35:head 2023-10-30T20:33:53.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:40b74ccd:::36:head 2023-10-30T20:33:53.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:4d5aaf42:::37:head 2023-10-30T20:33:53.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:920f362c:::38:head 2023-10-30T20:33:53.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6cc53222:::39:head 2023-10-30T20:33:53.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:9cad833f:::40:head 2023-10-30T20:33:53.360 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:1ea84d41:::41:head 2023-10-30T20:33:53.360 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c4480ef6:::42:head 2023-10-30T20:33:53.360 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a694361e:::43:head 2023-10-30T20:33:53.360 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d1bd33e9:::44:head 2023-10-30T20:33:53.361 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:ddc2cd5d:::45:head 2023-10-30T20:33:53.361 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:2b782207:::46:head 2023-10-30T20:33:53.361 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:7b187fca:::47:head 2023-10-30T20:33:53.361 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:90ecdf6f:::48:head 2023-10-30T20:33:53.362 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a5ed95fe:::49:head 2023-10-30T20:33:53.362 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:ea0eaa55:::50:head 2023-10-30T20:33:53.362 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f33ef17b:::51:head 2023-10-30T20:33:53.362 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a0d1b2f6:::52:head 2023-10-30T20:33:53.363 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:60c5229e:::53:head 2023-10-30T20:33:53.363 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:edcbc575:::54:head 2023-10-30T20:33:53.363 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:102cf253:::55:head 2023-10-30T20:33:53.363 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:efb7fb0b:::56:head 2023-10-30T20:33:53.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:50d0a326:::57:head 2023-10-30T20:33:53.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d4dc5daf:::58:head 2023-10-30T20:33:53.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:3a130462:::59:head 2023-10-30T20:33:53.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:ec87ed71:::60:head 2023-10-30T20:33:53.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d5bc9454:::61:head 2023-10-30T20:33:53.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:3ddfe313:::62:head 2023-10-30T20:33:53.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:7c2816b9:::63:head 2023-10-30T20:33:53.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:47e00e4d:::64:head 2023-10-30T20:33:53.366 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c6410c18:::65:head 2023-10-30T20:33:53.366 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b48ed237:::66:head 2023-10-30T20:33:53.366 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:cd63ad31:::67:head 2023-10-30T20:33:53.366 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b179e92b:::68:head 2023-10-30T20:33:53.367 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:0d9f741a:::69:head 2023-10-30T20:33:53.367 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6d3352ae:::70:head 2023-10-30T20:33:53.367 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c6d5c19e:::71:head 2023-10-30T20:33:53.367 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:bc4729c3:::72:head 2023-10-30T20:33:53.368 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:77e930b9:::73:head 2023-10-30T20:33:53.368 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:0abeecfd:::74:head 2023-10-30T20:33:53.368 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b7c37e15:::75:head 2023-10-30T20:33:53.368 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b6378398:::76:head 2023-10-30T20:33:53.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:02bd68de:::77:head 2023-10-30T20:33:53.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:cc795d2d:::78:head 2023-10-30T20:33:53.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:630d4fea:::79:head 2023-10-30T20:33:53.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:e0d29ef5:::80:head 2023-10-30T20:33:53.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:fd6f13d2:::81:head 2023-10-30T20:33:53.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:606461d5:::82:head 2023-10-30T20:33:53.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:eadbdc43:::83:head 2023-10-30T20:33:53.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:8761d0bb:::84:head 2023-10-30T20:33:53.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:9ef0186f:::85:head 2023-10-30T20:33:53.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:e0d41294:::86:head 2023-10-30T20:33:53.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:961de695:::87:head 2023-10-30T20:33:53.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:1423148f:::88:head 2023-10-30T20:33:53.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:633a8fa2:::89:head 2023-10-30T20:33:53.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a8653809:::90:head 2023-10-30T20:33:53.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:3dac8b33:::91:head 2023-10-30T20:33:53.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:35aad435:::92:head 2023-10-30T20:33:53.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f6dcc343:::93:head 2023-10-30T20:33:53.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:dbbdad87:::94:head 2023-10-30T20:33:53.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:1cb48ce0:::95:head 2023-10-30T20:33:53.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:03cd461c:::96:head 2023-10-30T20:33:53.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:17a4ea99:::97:head 2023-10-30T20:33:53.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:9993c9a7:::98:head 2023-10-30T20:33:53.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6394211c:::99:head 2023-10-30T20:33:53.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:94c7ae57:::100:head 2023-10-30T20:33:53.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6fdee5bb:::101:head 2023-10-30T20:33:53.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:9a477fd1:::102:head 2023-10-30T20:33:53.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:eb850916:::103:head 2023-10-30T20:33:53.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:affc56b9:::104:head 2023-10-30T20:33:53.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b42dc814:::105:head 2023-10-30T20:33:53.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f319f8f0:::106:head 2023-10-30T20:33:53.376 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:9a40b9de:::107:head 2023-10-30T20:33:53.376 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:8b524f28:::108:head 2023-10-30T20:33:53.376 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:e3de589f:::109:head 2023-10-30T20:33:53.376 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:90f90a5b:::110:head 2023-10-30T20:33:53.377 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a7b4f1d7:::111:head 2023-10-30T20:33:53.377 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:af51766e:::112:head 2023-10-30T20:33:53.377 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b6f90bd1:::113:head 2023-10-30T20:33:53.377 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:e0261208:::114:head 2023-10-30T20:33:53.378 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c9569ef7:::115:head 2023-10-30T20:33:53.378 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:61bebe50:::116:head 2023-10-30T20:33:53.378 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:fe93412b:::117:head 2023-10-30T20:33:53.378 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d3d38bee:::118:head 2023-10-30T20:33:53.379 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:3100ba0c:::119:head 2023-10-30T20:33:53.379 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d0560ada:::120:head 2023-10-30T20:33:53.379 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f0ea8b35:::121:head 2023-10-30T20:33:53.379 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:766f231a:::122:head 2023-10-30T20:33:53.379 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a07a2582:::123:head 2023-10-30T20:33:53.380 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:bd7c6b3a:::124:head 2023-10-30T20:33:53.380 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:fb2ddaff:::125:head 2023-10-30T20:33:53.380 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:4408e1fe:::126:head 2023-10-30T20:33:53.381 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:ee1df7a7:::127:head 2023-10-30T20:33:53.381 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c3002909:::128:head 2023-10-30T20:33:53.381 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:4f48ffa9:::129:head 2023-10-30T20:33:53.381 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:edf38733:::130:head 2023-10-30T20:33:53.381 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c08425c0:::131:head 2023-10-30T20:33:53.382 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:5f902d98:::132:head 2023-10-30T20:33:53.382 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:41ea2c93:::133:head 2023-10-30T20:33:53.382 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:813cee13:::134:head 2023-10-30T20:33:53.382 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:0131818d:::135:head 2023-10-30T20:33:53.383 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:26ba5a85:::136:head 2023-10-30T20:33:53.383 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:381b8a5a:::137:head 2023-10-30T20:33:53.383 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:28797e47:::138:head 2023-10-30T20:33:53.383 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:bfca7f22:::139:head 2023-10-30T20:33:53.384 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:36807075:::140:head 2023-10-30T20:33:53.384 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:80b03975:::141:head 2023-10-30T20:33:53.384 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:5c15709b:::142:head 2023-10-30T20:33:53.384 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f39ea15e:::143:head 2023-10-30T20:33:53.385 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:ea992956:::144:head 2023-10-30T20:33:53.385 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:48887b1c:::145:head 2023-10-30T20:33:53.385 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:9f24a9dd:::146:head 2023-10-30T20:33:53.385 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:987f100b:::147:head 2023-10-30T20:33:53.385 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d2dd3581:::148:head 2023-10-30T20:33:53.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:7fed1808:::149:head 2023-10-30T20:33:53.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c80b70e9:::150:head 2023-10-30T20:33:53.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:85ed90f9:::151:head 2023-10-30T20:33:53.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:36428b24:::152:head 2023-10-30T20:33:53.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d044c34a:::153:head 2023-10-30T20:33:53.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:7c18bf58:::154:head 2023-10-30T20:33:53.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d1c21232:::155:head 2023-10-30T20:33:53.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a7a3c575:::156:head 2023-10-30T20:33:53.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:87da0633:::157:head 2023-10-30T20:33:53.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d5ac3822:::158:head 2023-10-30T20:33:53.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:3f20522d:::159:head 2023-10-30T20:33:53.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6ca26563:::160:head 2023-10-30T20:33:53.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:532ce135:::161:head 2023-10-30T20:33:53.389 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c78863e6:::162:head 2023-10-30T20:34:12.366 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:34.424 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:12.368 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:34.426 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:12.398 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:34.456 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:12.411 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:34.438 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:12.412 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:34.471 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:12.413 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:34.440 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:12.416 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:34.475 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:12.439 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:34.466 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:12.441 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:34.500 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:13.150 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:35.176 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:13.151 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:35.177 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:13.163 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:35.190 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:13.208 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:35.267 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:13.209 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:35.268 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:13.228 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:35.286 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:13.228 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:35.287 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:13.229 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:35.288 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:13.242 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:35.300 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:32.285 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:54.343 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:32.291 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:54.349 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:32.328 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:54.386 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:32.334 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:54.361 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:32.337 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:54.364 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:32.347 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:54.405 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:32.350 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:54.409 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:32.364 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:54.391 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:32.377 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:54.435 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:52.366 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:14.424 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:52.370 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:14.428 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:52.402 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:14.461 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:52.419 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:14.446 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:52.420 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:14.479 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:52.421 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:14.447 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:52.422 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:14.480 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:52.448 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:14.475 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:52.451 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:14.510 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:57.375 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:19.433 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:57.379 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:19.437 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:57.391 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:19.449 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:57.427 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:19.486 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:57.428 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:19.455 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:57.429 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:19.456 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:57.430 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:19.489 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:57.438 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:19.465 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:34:57.440 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:19.499 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:02.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:7462ddf6:::163:head 2023-10-30T20:35:02.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:5d165639:::164:head 2023-10-30T20:35:02.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f43765fc:::165:head 2023-10-30T20:35:02.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b4c720e9:::166:head 2023-10-30T20:35:02.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:e694b040:::167:head 2023-10-30T20:35:02.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:afa38db2:::168:head 2023-10-30T20:35:02.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:77ba9f53:::169:head 2023-10-30T20:35:02.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:87495034:::170:head 2023-10-30T20:35:02.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:7c96bf0e:::171:head 2023-10-30T20:35:02.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:dbe346cc:::172:head 2023-10-30T20:35:02.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:e943ec24:::173:head 2023-10-30T20:35:02.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f97a9c0c:::174:head 2023-10-30T20:35:02.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6f26e74d:::175:head 2023-10-30T20:35:02.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:4f95e106:::176:head 2023-10-30T20:35:02.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:0e6f2f8f:::177:head 2023-10-30T20:35:02.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:05db05f1:::178:head 2023-10-30T20:35:02.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:38a78d66:::179:head 2023-10-30T20:35:02.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d095610b:::180:head 2023-10-30T20:35:02.928 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a1a9d709:::181:head 2023-10-30T20:35:02.928 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:1e5d39db:::182:head 2023-10-30T20:35:02.928 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f7df4fb9:::183:head 2023-10-30T20:35:02.928 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:03a7f161:::184:head 2023-10-30T20:35:02.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:ba70721e:::185:head 2023-10-30T20:35:02.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:28e5662d:::186:head 2023-10-30T20:35:02.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:973d52de:::187:head 2023-10-30T20:35:02.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:4303eb1c:::188:head 2023-10-30T20:35:02.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b990b48e:::189:head 2023-10-30T20:35:02.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:29b8165b:::190:head 2023-10-30T20:35:02.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:3547f197:::191:head 2023-10-30T20:35:02.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:7e260936:::192:head 2023-10-30T20:35:02.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:1abec7b1:::193:head 2023-10-30T20:35:02.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:10fdda93:::194:head 2023-10-30T20:35:02.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:15817eea:::195:head 2023-10-30T20:35:02.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:770bab57:::196:head 2023-10-30T20:35:02.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:ed9e13e7:::197:head 2023-10-30T20:35:02.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:71471a8f:::198:head 2023-10-30T20:35:02.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:10fb1d02:::199:head 2023-10-30T20:35:02.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (21358 ms) 2023-10-30T20:35:02.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2023-10-30T20:35:02.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698051,0 2023-10-30T20:35:02.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first is 1698698051 2023-10-30T20:35:02.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698051,0 2023-10-30T20:35:02.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698051,0 2023-10-30T20:35:02.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698051,1698698053,1698698054,0 2023-10-30T20:35:02.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698051,1698698053,1698698054,0 2023-10-30T20:35:02.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698051,1698698053,1698698054,0 2023-10-30T20:35:02.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698051,1698698053,1698698054,1698698056,1698698057,0 2023-10-30T20:35:02.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698051,1698698053,1698698054,1698698056,1698698057,0 2023-10-30T20:35:02.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698051,1698698053,1698698054,1698698056,1698698057,0 2023-10-30T20:35:02.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698051,1698698053,1698698054,1698698056,1698698057,1698698059,1698698060,0 2023-10-30T20:35:02.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698051,1698698053,1698698054,1698698056,1698698057,1698698059,1698698060,0 2023-10-30T20:35:02.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698051,1698698053,1698698054,1698698056,1698698057,1698698059,1698698060,0 2023-10-30T20:35:02.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698054,1698698056,1698698057,1698698059,1698698060,1698698063,1698698064,0 2023-10-30T20:35:02.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first now 1698698054, trimmed 2023-10-30T20:35:02.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (23003 ms) 2023-10-30T20:35:02.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2023-10-30T20:35:02.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: foo0 2023-10-30T20:35:02.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-10-30T20:35:02.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (15687 ms) 2023-10-30T20:35:02.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2023-10-30T20:35:12.468 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:34.526 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:12.471 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:34.530 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:12.498 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:34.556 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:12.509 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:34.536 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:12.512 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:34.539 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:12.514 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:34.572 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:12.517 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:34.576 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:12.537 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:34.564 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:12.544 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:34.603 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:13.267 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:35.293 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:13.270 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:35.296 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:13.282 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:35.309 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:13.335 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:35.394 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:13.336 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:35.395 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:13.351 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:35.410 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:13.352 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:35.411 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:13.353 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:35.412 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:13.364 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:35.422 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:27.495 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:49.553 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:27.496 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:49.554 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:27.515 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:49.574 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:27.541 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:49.568 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:27.543 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:49.602 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:27.544 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:49.571 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:27.545 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:49.604 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:27.561 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:49.588 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:27.564 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:49.622 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:32.602 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:54.660 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:32.688 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:54.714 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:32.689 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:54.735 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:32.701 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:54.728 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:32.704 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:54.731 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:32.718 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:54.744 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:32.722 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:54.781 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:32.725 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:54.783 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:32.743 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:54.801 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:42.601 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:04.659 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:42.606 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:04.664 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:42.630 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:04.688 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:42.651 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:04.678 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:42.653 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:04.680 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:42.670 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:04.729 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:42.672 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:04.731 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:42.675 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:04.702 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:42.694 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:04.752 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:52.634 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:14.692 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:52.637 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:14.695 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:52.658 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:14.716 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:52.674 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:14.701 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:52.677 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:14.704 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:52.699 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:14.757 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:52.699 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:14.726 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:52.701 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:14.759 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:52.723 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:14.782 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:57.642 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:19.700 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:57.646 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:19.704 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:57.661 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:19.719 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:57.690 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:19.748 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:57.692 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:19.719 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:57.693 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:19.720 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:57.695 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:19.754 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:57.708 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:19.735 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:35:57.710 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:19.769 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:12.665 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:34.722 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:12.713 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:34.740 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:12.716 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:34.743 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:12.719 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:34.777 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:12.738 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:34.765 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:12.744 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:34.803 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:12.764 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:34.823 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:12.769 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:34.828 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:12.791 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:34.850 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:13.338 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:35.365 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:13.339 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:35.366 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:13.346 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:35.373 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:13.387 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:35.445 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:13.388 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:35.446 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:13.396 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:35.454 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:13.422 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:35.480 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:13.423 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:35.481 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:13.431 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:35.489 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:27.787 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:49.844 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:27.870 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:49.928 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:27.897 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:49.955 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:27.919 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:49.946 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:27.921 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:49.948 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:27.926 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:49.985 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:27.931 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:49.990 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:27.939 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:49.966 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:27.949 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:50.008 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:32.914 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:54.972 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:32.915 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:54.973 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:32.935 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:54.994 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:32.965 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:54.992 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:32.966 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:54.993 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:32.986 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:55.013 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:32.987 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:55.046 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:32.989 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:55.047 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:33.058 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:55.117 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:33.512 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:55.538 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:33.512 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:55.540 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:33.514 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:55.572 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:33.514 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:55.573 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:33.520 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:13:55.547 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:33.523 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:55.581 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:33.582 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:55.641 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:33.583 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:55.642 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:33.591 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:55.650 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:42.855 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:04.911 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:42.899 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:04.926 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:42.900 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:04.958 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:42.901 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:04.959 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:42.901 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:04.928 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:42.904 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:04.962 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:42.922 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:04.949 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:42.929 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:04.988 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:42.930 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:04.989 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:58.060 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:20.118 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:58.131 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:20.189 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:58.164 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:20.222 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:58.172 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:20.230 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:58.194 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:20.221 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:58.196 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:20.222 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:58.220 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:20.247 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:58.263 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:20.322 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:36:58.317 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:20.376 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:13.002 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:35.061 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:13.006 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:35.065 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:13.029 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:35.087 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:13.055 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:35.082 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:13.055 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:35.114 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:13.058 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:35.085 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:13.059 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:35.117 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:13.080 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:35.139 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:13.083 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:35.110 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:18.008 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:40.067 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:18.014 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:40.072 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:18.023 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:40.082 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:18.056 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:40.115 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:18.056 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:40.083 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:18.059 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:40.118 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:18.060 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:40.086 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:18.069 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:40.097 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:18.070 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:40.129 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:33.173 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:55.230 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:33.176 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:55.235 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:33.199 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:55.258 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:33.221 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:55.248 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:33.222 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:55.249 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:33.223 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:55.282 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:33.225 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:55.284 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:33.240 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:14:55.268 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:33.245 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:55.303 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:38.083 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:00.141 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:38.086 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:00.144 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:38.098 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:00.157 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:38.135 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:00.194 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:38.136 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:00.163 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:38.137 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:00.195 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:38.137 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:00.164 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:38.147 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:00.206 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:38.148 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:00.174 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:48.106 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:10.164 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:48.109 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:10.167 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:48.125 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:10.184 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:48.154 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:10.181 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:48.156 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:10.183 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:48.158 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:10.216 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:48.159 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:10.218 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:48.172 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:10.199 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:48.175 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:10.234 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:58.215 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:20.273 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:58.283 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:20.342 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:58.312 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:20.371 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:58.370 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:20.429 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:58.375 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:20.434 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:58.405 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:20.464 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:58.411 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:20.439 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:58.416 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:20.443 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:37:58.450 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:20.477 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:03.114 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:25.172 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:03.173 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:25.232 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:03.187 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:25.245 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:03.201 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:25.228 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:03.204 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:25.231 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:03.205 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:25.264 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:03.209 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:25.268 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:03.217 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:25.244 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:03.221 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:25.280 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.155 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:40.213 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.202 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:40.260 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.218 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:40.277 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.248 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:40.307 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.248 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:40.276 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.250 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:40.309 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.251 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:40.278 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.266 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:40.325 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.267 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:40.294 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.699 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:40.726 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.700 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:40.728 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.706 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:40.764 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.707 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:40.765 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.710 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:40.737 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.719 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:40.777 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.729 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:40.787 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.730 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:40.788 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:18.737 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:40.796 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:21.158 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-10-30T20:38:21.138+0000 22a89640 -1 snap_mapper.add_oid found existing snaps mapped on 219:2cb6d319:test-rados-api-smithi069-91800-80::foo:2, removing 2023-10-30T20:38:21.163 INFO:tasks.ceph.osd.3.smithi069.stderr:2023-10-30T20:38:21.153+0000 27292640 -1 snap_mapper.add_oid found existing snaps mapped on 219:2cb6d319:test-rados-api-smithi069-91800-80::foo:2, removing 2023-10-30T20:38:33.252 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:55.311 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:33.256 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:55.315 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:33.283 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:55.341 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:33.300 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:55.327 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:33.302 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:55.329 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:33.304 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:55.363 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:33.308 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:55.367 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:33.328 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:15:55.356 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:33.335 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:55.394 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:38.262 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:00.320 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:38.266 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:00.325 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:38.290 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:00.348 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:38.311 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:00.337 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:38.313 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:00.340 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:38.320 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:00.378 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:38.321 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:00.380 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:38.326 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:00.353 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:38.334 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:00.392 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:47.425 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:09.483 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:47.430 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:09.488 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:47.440 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:09.467 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:47.442 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:09.500 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:47.442 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:09.469 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:47.446 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:09.505 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:47.525 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:09.552 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:47.551 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:09.609 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:47.559 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:09.617 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:48.419 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:10.477 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:48.427 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:10.486 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:48.468 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:10.527 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:48.479 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:10.506 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:48.486 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:10.513 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:48.490 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:10.549 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:48.496 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:10.554 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:48.497 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:10.524 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:48.507 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:10.566 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:58.338 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:20.394 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:58.400 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:20.458 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:58.463 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:20.521 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:58.464 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:20.522 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:58.480 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:20.538 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:58.512 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:20.539 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:58.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (21626 ms) 2023-10-30T20:38:58.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2023-10-30T20:38:58.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (25217 ms) 2023-10-30T20:38:58.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2023-10-30T20:38:58.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4455 ms) 2023-10-30T20:38:58.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2023-10-30T20:38:58.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (4448 ms) 2023-10-30T20:38:58.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2023-10-30T20:38:58.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3820 ms) 2023-10-30T20:38:58.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2023-10-30T20:38:58.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4630 ms) 2023-10-30T20:38:58.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2023-10-30T20:38:58.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5582 ms) 2023-10-30T20:38:58.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2023-10-30T20:38:58.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (41623 ms) 2023-10-30T20:38:58.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2023-10-30T20:38:58.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (18482 ms) 2023-10-30T20:38:58.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2023-10-30T20:38:58.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5367 ms) 2023-10-30T20:38:58.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2023-10-30T20:38:58.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4289 ms) 2023-10-30T20:38:58.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2023-10-30T20:38:58.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (26927 ms) 2023-10-30T20:38:58.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2023-10-30T20:38:58.524 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.524 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16241 ms) 2023-10-30T20:38:58.524 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2023-10-30T20:38:58.525 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.525 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6756 ms) 2023-10-30T20:38:58.525 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2023-10-30T20:38:58.525 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5512 ms) 2023-10-30T20:38:58.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2023-10-30T20:38:58.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrubs... 2023-10-30T20:38:58.527 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2023-10-30T20:38:58.527 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26412 ms) 2023-10-30T20:38:58.527 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2023-10-30T20:38:58.527 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.528 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (11331 ms) 2023-10-30T20:38:58.528 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2023-10-30T20:38:58.528 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.528 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (11147 ms) 2023-10-30T20:38:58.529 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2023-10-30T20:38:58.529 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:38:58.529 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (10796 ms) 2023-10-30T20:38:58.529 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2023-10-30T20:38:58.530 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:20.548 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:58.530 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:20.579 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:38:58.549 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:20.576 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:03.390 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:25.448 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:03.391 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:25.450 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:03.410 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:25.468 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:03.439 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:25.466 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:03.441 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:25.468 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:03.445 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:25.504 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:03.447 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:25.506 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:03.456 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:25.483 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:03.466 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:25.524 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:18.457 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:40.510 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:18.460 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:40.515 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:18.498 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:40.525 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:18.500 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:40.527 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:18.505 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:40.564 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:18.508 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:40.566 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:18.509 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:40.568 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:18.527 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:40.554 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:18.541 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:40.600 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:19.071 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:41.097 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:19.071 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:41.098 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:19.078 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:41.105 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:19.118 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:41.177 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:19.119 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:41.177 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:19.119 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:41.178 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:19.122 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:41.180 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:19.128 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:41.187 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:19.131 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:41.190 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:33.503 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:55.560 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:33.505 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:55.563 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:33.523 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:55.581 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:33.548 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:55.606 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:33.550 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:55.608 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:33.554 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:55.581 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:33.556 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:55.583 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:33.571 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:55.630 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:33.575 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:16:55.602 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:38.646 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:00.704 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:38.652 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:00.710 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:38.667 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:00.725 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:38.700 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:00.726 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:38.705 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:00.732 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:38.706 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:00.764 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:38.712 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:00.771 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:38.722 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:00.749 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:38.730 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:00.788 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:48.683 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:10.741 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:48.689 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:10.747 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:48.713 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:10.771 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:48.741 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:10.768 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:48.744 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:10.771 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:48.753 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:10.812 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:48.755 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:10.814 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:48.765 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:10.792 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:48.827 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:10.885 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:53.781 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:15.840 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:53.784 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:15.842 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:53.788 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:15.815 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:53.806 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:15.864 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:53.837 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:15.864 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:53.845 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:15.903 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:53.849 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:15.908 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:53.853 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:15.880 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:53.872 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:15.931 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:58.706 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:20.764 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:58.711 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:20.769 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:58.722 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:20.781 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:58.759 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:20.817 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:58.759 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:20.786 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:58.760 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:20.787 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:58.760 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:20.819 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:58.771 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:20.798 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:39:58.772 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:20.831 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:02.289 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-10-30T20:40:02.254+0000 22a89640 -1 snap_mapper.add_oid found existing snaps mapped on 78:c712c6b0:test-rados-api-smithi069-91800-97::foo:31, removing 2023-10-30T20:40:02.301 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-10-30T20:40:02.295+0000 2aa99640 -1 snap_mapper.add_oid found existing snaps mapped on 78:c712c6b0:test-rados-api-smithi069-91800-97::foo:31, removing 2023-10-30T20:40:08.709 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:30.766 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:08.754 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:30.813 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:08.773 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:30.831 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:08.798 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:30.825 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:08.802 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:30.829 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:08.805 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:30.864 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:08.808 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:30.867 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:08.819 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:30.847 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:08.826 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:30.885 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:23.918 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:45.977 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:23.964 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:45.992 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:23.965 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:46.023 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:23.967 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:45.994 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:23.999 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:17:46.027 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:24.005 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:46.063 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:24.028 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:46.087 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:24.034 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:46.093 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:24.065 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:46.124 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:38.941 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:00.999 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:38.944 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:01.003 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:39.010 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:01.038 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:39.012 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:01.039 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:39.034 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:01.093 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:39.048 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:01.075 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:39.106 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:01.165 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:39.109 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:01.167 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:39.139 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:01.198 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:43.892 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:05.950 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:43.897 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:05.956 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:43.917 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:05.976 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:43.941 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:05.968 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:43.943 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:05.970 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:43.955 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:05.982 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:43.959 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:06.018 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:43.961 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:06.019 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:43.973 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:06.032 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:52.203 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:14.262 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:52.207 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:14.265 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:52.216 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:14.244 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:52.220 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:14.279 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:52.223 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:14.282 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:52.269 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:14.296 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:52.304 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:14.363 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:52.308 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:14.367 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:52.313 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:14.340 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:54.066 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:16.124 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:54.105 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:16.163 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:54.115 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:16.143 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:54.118 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:16.145 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:54.128 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:16.155 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:54.136 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:16.195 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:54.137 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:16.196 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:54.143 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:16.202 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:40:54.170 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:16.228 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:04.058 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:26.116 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:04.061 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:26.119 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:04.089 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:26.147 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:04.112 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:26.139 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:04.113 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:26.140 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:04.133 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:26.191 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:04.134 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:26.193 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:04.139 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:26.165 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:04.157 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:26.216 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:08.976 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:31.030 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:08.981 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:31.032 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:08.986 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:31.044 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:09.019 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:31.046 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:09.020 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:31.047 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:09.022 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:31.081 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:09.023 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:31.082 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:09.032 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:31.059 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:09.036 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:31.094 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:24.139 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:46.194 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:24.141 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:46.197 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:24.146 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:46.173 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:24.147 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:46.206 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:24.150 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:46.177 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:24.152 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:46.211 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:24.236 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:46.295 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:24.238 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:46.297 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:24.242 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:46.269 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:29.050 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:51.107 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:29.096 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:51.155 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:29.102 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:51.129 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:29.104 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:51.131 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:29.113 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:51.171 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:29.124 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:18:51.151 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:29.145 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:51.203 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:29.149 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:51.208 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:29.164 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:51.223 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:44.231 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:06.288 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:44.234 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:06.291 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:44.261 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:06.319 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:44.277 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:19:06.305 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:44.280 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:19:06.307 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:44.300 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:06.359 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:44.301 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:06.360 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:44.309 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:19:06.336 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:41:44.332 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:06.391 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:04.270 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:26.325 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:04.280 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:26.339 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:04.291 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:19:26.318 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:04.292 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:26.351 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:04.296 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:26.355 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:04.348 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:19:26.375 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:04.374 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:19:26.400 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:04.380 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:26.439 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:04.419 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:26.478 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:09.274 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:31.332 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:09.325 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:31.384 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:09.343 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:31.401 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:09.370 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:19:31.398 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:09.375 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:19:31.402 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:09.379 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:31.438 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:09.380 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:31.439 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:09.388 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:19:31.415 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:09.395 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:31.454 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:24.427 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:46.485 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:24.489 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:46.548 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:24.532 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:46.590 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:24.556 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:19:46.584 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:24.561 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:19:46.588 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:24.564 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:46.623 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:24.568 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:46.627 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:24.591 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:19:46.618 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:24.606 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:46.665 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:39.525 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:01.583 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:39.526 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:01.585 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:39.558 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:01.617 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:39.573 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:01.600 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:39.575 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:01.602 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:39.577 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:01.636 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:39.579 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:01.638 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:39.602 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:01.629 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:39.612 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:01.671 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:44.438 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:06.497 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:44.439 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:06.498 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:44.452 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:06.510 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:44.481 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:06.509 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:44.484 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:06.512 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:44.500 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:06.528 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:44.503 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:06.562 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:44.507 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:06.566 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:44.522 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:06.581 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:48.488 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:10.547 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:48.492 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:10.551 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:48.508 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:10.567 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:48.540 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:10.567 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:48.541 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:10.569 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:48.542 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:10.602 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:48.544 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:10.604 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:48.557 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:10.585 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:48.561 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:10.620 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:53.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:42:53.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (11251 ms) 2023-10-30T20:42:53.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2023-10-30T20:42:53.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:42:53.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (7726 ms) 2023-10-30T20:42:53.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2023-10-30T20:42:53.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:42:53.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6835 ms) 2023-10-30T20:42:53.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2023-10-30T20:42:53.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:42:53.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26240 ms) 2023-10-30T20:42:53.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2023-10-30T20:42:53.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:42:53.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15653 ms) 2023-10-30T20:42:53.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2023-10-30T20:42:53.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (13485 ms) 2023-10-30T20:42:53.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2023-10-30T20:42:53.980 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2023-10-30T20:42:53.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000010 69 73 |is| 2023-10-30T20:42:53.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000012 2023-10-30T20:42:53.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12730 ms) 2023-10-30T20:42:53.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2023-10-30T20:42:53.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T20:42:53.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6588 ms) 2023-10-30T20:42:53.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (786733 ms total) 2023-10-30T20:42:53.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2023-10-30T20:42:53.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2023-10-30T20:42:53.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2023-10-30T20:42:53.984 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1578 ms) 2023-10-30T20:42:53.984 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2023-10-30T20:42:53.984 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11293 ms) 2023-10-30T20:42:53.985 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2023-10-30T20:42:53.985 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18596 ms) 2023-10-30T20:42:53.985 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2023-10-30T20:42:53.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (55 ms) 2023-10-30T20:42:53.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (31522 ms total) 2023-10-30T20:42:53.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2023-10-30T20:42:53.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2023-10-30T20:42:53.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2023-10-30T20:42:53.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7836 ms) 2023-10-30T20:42:53.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2023-10-30T20:42:53.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (9113 ms) 2023-10-30T20:42:53.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2023-10-30T20:42:53.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrub... 2023-10-30T20:42:53.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2023-10-30T20:42:53.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (27681 ms) 2023-10-30T20:42:53.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2023-10-30T20:42:53.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10533 ms) 2023-10-30T20:42:53.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2023-10-30T20:42:53.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (9972 ms) 2023-10-30T20:42:53.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2023-10-30T20:42:53.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (9027 ms) 2023-10-30T20:42:53.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2023-10-30T20:42:53.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (11166 ms) 2023-10-30T20:42:53.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2023-10-30T20:42:54.626 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:16.684 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:54.688 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:16.715 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:54.690 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:16.717 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:54.696 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:16.754 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:54.715 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:16.742 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:54.733 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:16.792 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:54.748 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:16.806 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:54.755 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:16.814 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:42:54.779 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:16.838 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:04.561 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:26.619 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:04.564 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:26.623 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:04.565 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:26.592 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:04.569 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:26.596 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:04.586 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:26.645 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:04.589 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:26.616 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:04.610 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:26.669 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:04.613 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:26.671 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:04.630 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:26.689 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:09.573 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:31.632 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:09.577 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:31.636 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:09.593 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:31.652 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:09.623 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:31.650 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:09.624 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:31.651 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:09.624 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:31.683 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:09.627 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:31.686 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:09.636 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:31.663 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:09.642 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:31.700 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:24.745 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:46.803 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:24.748 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:46.806 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:24.753 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:46.781 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:24.757 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:46.816 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:24.804 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:46.831 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:24.810 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:46.869 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:24.842 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:46.869 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:24.846 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:46.904 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:24.848 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:46.907 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:29.823 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:51.881 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:29.827 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:51.854 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:29.828 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:51.886 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:29.832 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:51.859 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:29.843 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:51.902 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:29.845 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:20:51.872 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:29.871 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:51.930 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:29.873 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:51.932 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:29.887 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:51.946 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:39.912 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:01.971 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:39.913 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:01.972 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:39.944 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:02.003 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:39.962 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:01.989 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:39.963 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:01.991 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:39.965 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:02.024 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:39.969 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:02.028 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:39.986 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:02.013 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:39.994 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:02.053 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:49.796 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:11.855 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:49.799 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:11.858 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:49.831 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:11.890 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:49.855 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:11.882 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:49.858 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:11.885 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:49.858 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:11.917 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:49.860 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:11.919 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:49.881 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:11.908 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:49.882 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:11.941 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:59.775 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:21.834 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:59.820 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:21.847 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:59.821 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:21.849 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:59.824 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:21.883 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:59.826 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:21.885 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:59.826 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:21.885 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:59.843 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:21.870 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:59.855 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:21.914 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:43:59.858 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:21.917 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:09.862 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:31.920 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:09.865 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:31.923 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:09.884 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:31.942 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:09.909 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:31.968 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:09.912 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:31.971 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:09.913 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:31.940 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:09.914 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:31.941 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:09.933 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:31.960 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:09.934 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:31.992 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:14.920 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:36.978 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:14.922 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:36.981 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:14.936 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:36.995 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:14.967 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:37.026 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:14.969 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:36.996 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:14.970 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:36.998 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:14.971 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:37.030 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:14.982 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:37.010 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:14.983 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:37.043 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:30.052 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:52.077 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:30.053 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:52.080 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:30.065 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:52.123 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:30.068 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:52.126 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:30.075 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:21:52.102 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:30.089 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:52.147 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:30.140 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:52.199 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:30.144 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:52.203 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:30.166 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:52.225 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:45.187 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:07.244 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:45.192 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:07.251 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:45.233 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:07.291 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:45.241 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:07.299 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:45.255 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:07.283 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:45.266 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:07.293 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:45.273 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:07.332 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:45.314 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:07.341 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:45.315 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:07.374 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:50.094 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:12.152 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:50.098 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:12.157 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:50.113 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:12.170 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:50.141 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:12.169 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:50.143 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:12.170 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:50.143 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:12.202 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:50.145 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:12.204 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:50.156 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:12.183 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:50.158 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:12.217 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:51.200 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:13.258 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:51.202 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:13.261 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:51.216 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:13.275 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:51.251 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:13.279 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:51.252 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:13.280 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:51.256 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:13.315 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:51.257 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:13.316 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:51.264 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:13.292 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:44:51.271 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:13.330 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:00.034 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:22.092 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:00.037 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:22.095 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:00.048 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:22.107 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:00.085 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:22.144 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:00.085 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:22.112 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:00.086 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:22.145 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:00.086 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:22.114 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:00.096 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:22.155 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:00.097 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:22.124 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:10.245 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:32.303 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:10.248 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:32.307 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:10.270 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:32.329 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:10.293 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:32.320 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:10.294 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:32.321 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:10.297 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:32.356 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:10.300 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:32.360 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:10.315 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:32.342 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:10.324 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:32.383 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:15.117 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:37.176 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:15.120 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:37.178 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:15.132 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:37.191 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:15.162 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:37.190 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:15.165 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:37.192 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:15.166 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:37.225 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:15.168 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:37.227 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:15.177 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:37.204 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:15.180 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:37.239 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:30.351 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:52.410 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:30.358 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:52.417 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:30.387 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:52.446 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:30.411 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:52.438 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:30.414 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:52.442 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:30.436 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:22:52.463 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:30.446 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:52.505 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:30.452 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:52.511 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:30.484 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:52.543 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:45.582 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:07.641 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:45.586 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:07.645 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:45.618 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:07.677 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:45.620 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:07.679 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:45.647 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:07.674 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:45.648 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:07.676 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:45.677 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:07.704 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:45.695 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:07.754 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:45.730 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:07.789 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:50.306 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:12.364 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:50.310 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:12.369 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:50.324 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:12.383 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:50.354 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:12.381 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:50.357 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:12.384 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:50.357 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:12.416 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:50.360 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:12.419 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:50.371 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:12.398 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:50.377 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:12.436 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:45:50.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (10164 ms) 2023-10-30T20:45:50.711 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2023-10-30T20:45:50.711 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14142 ms) 2023-10-30T20:45:50.711 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2023-10-30T20:45:50.711 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (10237 ms) 2023-10-30T20:45:50.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2023-10-30T20:45:50.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (15058 ms) 2023-10-30T20:45:50.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2023-10-30T20:45:50.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (10279 ms) 2023-10-30T20:45:50.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2023-10-30T20:45:50.712 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (10017 ms) 2023-10-30T20:45:50.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2023-10-30T20:45:50.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: hmm, no HitSet yet 2023-10-30T20:45:50.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: ok, hit_set contains 251:602f83fe:::foo:head 2023-10-30T20:45:50.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (10050 ms) 2023-10-30T20:45:50.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2023-10-30T20:45:50.714 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698652,0 2023-10-30T20:45:50.714 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first is 1698698652 2023-10-30T20:45:50.714 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698652,0 2023-10-30T20:45:50.714 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698652,0 2023-10-30T20:45:50.714 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698652,1698698654,1698698655,0 2023-10-30T20:45:50.715 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698652,1698698654,1698698655,0 2023-10-30T20:45:50.715 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698652,1698698654,1698698655,0 2023-10-30T20:45:50.715 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698652,1698698654,1698698655,1698698657,1698698658,0 2023-10-30T20:45:50.715 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698652,1698698654,1698698655,1698698657,1698698658,0 2023-10-30T20:45:50.715 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698652,1698698654,1698698655,1698698657,1698698658,0 2023-10-30T20:45:50.715 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698652,1698698654,1698698655,1698698657,1698698658,1698698661,1698698662,0 2023-10-30T20:45:50.716 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698652,1698698654,1698698655,1698698657,1698698658,1698698661,1698698662,0 2023-10-30T20:45:50.716 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698652,1698698654,1698698655,1698698657,1698698658,1698698661,1698698662,0 2023-10-30T20:45:50.716 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1698698655,1698698657,1698698658,1698698661,1698698662,1698698664,1698698665,0 2023-10-30T20:45:50.716 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first now 1698698655, trimmed 2023-10-30T20:45:50.716 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (23165 ms) 2023-10-30T20:45:50.717 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2023-10-30T20:45:50.717 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: foo0 2023-10-30T20:45:50.717 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-10-30T20:45:50.717 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (16645 ms) 2023-10-30T20:45:50.717 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2023-10-30T20:45:50.717 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (19905 ms) 2023-10-30T20:45:50.718 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2023-10-30T20:45:50.718 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (24441 ms) 2023-10-30T20:45:50.718 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2023-10-30T20:45:50.718 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (6010 ms) 2023-10-30T20:45:50.718 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2023-10-30T20:45:50.719 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4444 ms) 2023-10-30T20:45:50.719 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2023-10-30T20:45:50.719 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4505 ms) 2023-10-30T20:45:50.719 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2023-10-30T20:45:50.719 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4545 ms) 2023-10-30T20:45:50.720 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (268935 ms total) 2023-10-30T20:45:50.720 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2023-10-30T20:45:50.720 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] Global test environment tear-down 2023-10-30T20:45:50.720 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1233917 ms total) 2023-10-30T20:45:50.720 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ PASSED ] 77 tests. 2023-10-30T20:45:50.724 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.724 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91519 2023-10-30T20:45:50.724 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91519 2023-10-30T20:45:50.725 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.725 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91737 2023-10-30T20:45:50.725 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91737 2023-10-30T20:45:50.725 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.725 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92248 2023-10-30T20:45:50.726 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92248 2023-10-30T20:45:50.726 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.726 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91996 2023-10-30T20:45:50.726 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91996 2023-10-30T20:45:50.726 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.727 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91882 2023-10-30T20:45:50.727 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91882 2023-10-30T20:45:50.727 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.727 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91593 2023-10-30T20:45:50.727 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91593 2023-10-30T20:45:50.727 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.729 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91946 2023-10-30T20:45:50.729 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91946 2023-10-30T20:45:50.729 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.729 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91610 2023-10-30T20:45:50.730 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91610 2023-10-30T20:45:50.730 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.730 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92136 2023-10-30T20:45:50.730 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92136 2023-10-30T20:45:50.730 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.731 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91494 2023-10-30T20:45:50.731 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91494 2023-10-30T20:45:50.731 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.731 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91557 2023-10-30T20:45:50.731 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91557 2023-10-30T20:45:50.732 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.732 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92149 2023-10-30T20:45:50.732 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92149 2023-10-30T20:45:50.732 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.732 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91485 2023-10-30T20:45:50.732 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91485 2023-10-30T20:45:50.733 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.733 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91505 2023-10-30T20:45:50.733 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91505 2023-10-30T20:45:50.733 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.733 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91763 2023-10-30T20:45:50.734 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91763 2023-10-30T20:45:50.734 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.734 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91895 2023-10-30T20:45:50.734 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91895 2023-10-30T20:45:50.734 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.735 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91809 2023-10-30T20:45:50.735 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91809 2023-10-30T20:45:50.735 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.735 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92214 2023-10-30T20:45:50.735 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92214 2023-10-30T20:45:50.735 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.736 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92114 2023-10-30T20:45:50.736 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92114 2023-10-30T20:45:50.736 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.736 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91705 2023-10-30T20:45:50.736 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91705 2023-10-30T20:45:50.737 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.737 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92020 2023-10-30T20:45:50.737 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92020 2023-10-30T20:45:50.737 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.737 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91478 2023-10-30T20:45:50.737 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91478 2023-10-30T20:45:50.738 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.738 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91535 2023-10-30T20:45:50.738 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91535 2023-10-30T20:45:50.738 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.738 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91970 2023-10-30T20:45:50.738 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91970 2023-10-30T20:45:50.739 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.739 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91579 2023-10-30T20:45:50.739 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91579 2023-10-30T20:45:50.739 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.739 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92046 2023-10-30T20:45:50.739 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92046 2023-10-30T20:45:50.740 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-10-30T20:45:50.740 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92064 2023-10-30T20:45:50.740 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92064 2023-10-30T20:45:50.740 INFO:tasks.workunit.client.0.smithi069.stderr:+ exit 0 2023-10-30T20:45:50.740 INFO:tasks.workunit.client.0.smithi069.stderr:+ cleanup 2023-10-30T20:45:50.741 INFO:tasks.workunit.client.0.smithi069.stderr:+ pkill -P 91473 2023-10-30T20:45:50.741 INFO:tasks.workunit.client.0.smithi069.stderr:+ true 2023-10-30T20:45:50.742 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T20:45:50.742 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T20:45:50.771 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2023-10-30T20:45:50.771 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-30T20:45:51.093 DEBUG:teuthology.parallel:result is None 2023-10-30T20:45:51.094 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-10-30T20:45:51.120 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-10-30T20:45:51.121 DEBUG:teuthology.orchestra.run.smithi069:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-10-30T20:45:51.187 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-10-30T20:45:51.187 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-30T20:45:51.197 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T20:45:51.198 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T20:45:51.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.325+0000 7f909b8fc640 10 monclient: get_monmap_and_config 2023-10-30T20:45:51.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.325+0000 7f909b8fc640 10 monclient: build_initial_monmap 2023-10-30T20:45:51.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.325+0000 7f909b8fc640 10 monclient: monmap: 2023-10-30T20:45:51.327 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:51.327 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:51.327 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:51.326038+0000 2023-10-30T20:45:51.327 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:51.326038+0000 2023-10-30T20:45:51.328 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:51.328 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:51.328 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:51.328 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:51.328 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:51.329 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:51.329 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:51.329 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:51.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.325+0000 7f909b8fc640 1 Processor -- start 2023-10-30T20:45:51.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.325+0000 7f909b8fc640 1 -- start start 2023-10-30T20:45:51.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.325+0000 7f909b8fc640 10 monclient: init 2023-10-30T20:45:51.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.326+0000 7f909b8fc640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:51.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.326+0000 7f909b8fc640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-10-30T20:45:51.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.326+0000 7f909b8fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f90940aea90 addr v1:172.21.15.69:6789/0 2023-10-30T20:45:51.333 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.326+0000 7f909b8fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90940b12c0 0x7f90940ad520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:51.333 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.326+0000 7f909b8fc640 10 monclient(hunting): picked mon.noname-b con 0x7f90940b12c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:51.333 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.326+0000 7f909b8fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90940ada60 addr v1:172.21.15.183:6789/0 2023-10-30T20:45:51.333 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.326+0000 7f909b8fc640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90940b0e20 con 0x7f90940aea90 2023-10-30T20:45:51.333 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.326+0000 7f909b8fc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90940afe80 con 0x7f90940ada60 2023-10-30T20:45:51.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.326+0000 7f909b8fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:51.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.326+0000 7f909b8fc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90940679a0 con 0x7f90940b12c0 2023-10-30T20:45:51.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.326+0000 7f909b8fc640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:51.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.326+0000 7f909b8fc640 10 monclient(hunting): authenticate will time out at 2874.615413s 2023-10-30T20:45:51.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.327+0000 7f9098e97640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90940b12c0 0x7f90940ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:51.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.327+0000 7f9099698640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f90940aea90 0x7f9094141900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:38876/0 (socket says 172.21.15.69:38876) 2023-10-30T20:45:51.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.327+0000 7f9099698640 1 -- 172.21.15.69:0/1304558070 learned_addr learned my addr 172.21.15.69:0/1304558070 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:45:51.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.327+0000 7f9098e97640 10 monclient(hunting): get_auth_request con 0x7f90940b12c0 auth_method 0 2023-10-30T20:45:51.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.327+0000 7f9098e97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:51.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.327+0000 7f9098e97640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:51.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.327+0000 7f9098e97640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:51.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.328+0000 7f9098e97640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:51.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.328+0000 7f9098e97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:51.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.328+0000 7f9098e97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:51.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.329+0000 7f908bfff640 1 -- 172.21.15.69:0/1304558070 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859171198 0 0) 0x7f908c001490 con 0x7f90940ada60 2023-10-30T20:45:51.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.329+0000 7f908bfff640 10 monclient(hunting): _negotiate 2023-10-30T20:45:51.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.329+0000 7f908bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:51.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.329+0000 7f908bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:51.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.329+0000 7f908bfff640 10 monclient(hunting): my global_id is 21196 2023-10-30T20:45:51.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.329+0000 7f908bfff640 1 -- 172.21.15.69:0/1304558070 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9080003520 con 0x7f90940ada60 2023-10-30T20:45:51.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.329+0000 7f9098e97640 10 monclient(hunting): handle_auth_done global_id 18257 payload 274 2023-10-30T20:45:51.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.329+0000 7f9098e97640 1 -- 172.21.15.69:0/1304558070 >> v1:172.21.15.183:6789/0 conn(0x7f90940ada60 legacy=0x7f909414c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:51.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.329+0000 7f9098e97640 1 -- 172.21.15.69:0/1304558070 >> v1:172.21.15.69:6789/0 conn(0x7f90940aea90 legacy=0x7f9094141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:51.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.329+0000 7f9098e97640 10 monclient: _finish_hunting 0 2023-10-30T20:45:51.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.329+0000 7f9098e97640 1 monclient: found mon.noname-b 2023-10-30T20:45:51.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.329+0000 7f9098e97640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:51.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.329+0000 7f9098e97640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:45:51.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.329+0000 7f9098e97640 1 -- 172.21.15.69:0/1304558070 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90940b0160 con 0x7f90940b12c0 2023-10-30T20:45:51.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.331+0000 7f9098e97640 1 --2- 172.21.15.69:0/1304558070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90940b12c0 0x7f90940ad520 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7f90840097d0 tx=0x7f908402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8566d0d82cc38ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:51.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f908bfff640 1 -- 172.21.15.69:0/1304558070 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084031800 con 0x7f90940b12c0 2023-10-30T20:45:51.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f908bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:51.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f908bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:51.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f908bfff640 10 monclient: dump: 2023-10-30T20:45:51.346 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:51.346 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:51.346 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.347 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.347 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:51.347 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:51.347 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:51.347 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:51.348 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:51.348 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:51.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f908bfff640 10 monclient: _finish_auth 0 2023-10-30T20:45:51.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f908bfff640 10 monclient: _check_auth_tickets 2023-10-30T20:45:51.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f908bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:51.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f908bfff640 1 -- 172.21.15.69:0/1304558070 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9084031970 con 0x7f90940b12c0 2023-10-30T20:45:51.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f909b8fc640 5 monclient: authenticate success, global_id 18257 2023-10-30T20:45:51.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f909b8fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:45:51.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f908bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:51.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f909b8fc640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:51.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f908bfff640 1 -- 172.21.15.69:0/1304558070 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084031d10 con 0x7f90940b12c0 2023-10-30T20:45:51.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f909b8fc640 4 set_mon_vals no callback set 2023-10-30T20:45:51.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f908bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:51.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f908bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:45:51.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f908bfff640 10 monclient: dump: 2023-10-30T20:45:51.351 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:51.351 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:51.351 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.352 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.352 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:51.352 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:51.352 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:51.352 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:51.353 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:51.353 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:51.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f909b8fc640 10 monclient: shutdown 2023-10-30T20:45:51.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f909b8fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:51.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f909b8fc640 1 -- 172.21.15.69:0/1304558070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90940b12c0 msgr2=0x7f90940ad520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:51.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.335+0000 7f909b8fc640 1 --2- 172.21.15.69:0/1304558070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90940b12c0 0x7f90940ad520 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7f90840097d0 tx=0x7f908402b8a0 comp rx=0 tx=0).stop 2023-10-30T20:45:51.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.336+0000 7f909b8fc640 1 -- 172.21.15.69:0/1304558070 shutdown_connections 2023-10-30T20:45:51.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.336+0000 7f909b8fc640 1 --2- 172.21.15.69:0/1304558070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90940b12c0 0x7f90940ad520 unknown :-1 s=CLOSED pgs=3198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:51.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.336+0000 7f909b8fc640 1 -- 172.21.15.69:0/1304558070 >> 172.21.15.69:0/1304558070 conn(0x7f90940b7120 msgr2=0x7f90940b7500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:51.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.336+0000 7f909b8fc640 1 -- 172.21.15.69:0/1304558070 shutdown_connections 2023-10-30T20:45:51.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.336+0000 7f909b8fc640 1 -- 172.21.15.69:0/1304558070 wait complete. 2023-10-30T20:45:51.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.336+0000 7f909b8fc640 10 monclient: build_initial_monmap 2023-10-30T20:45:51.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.336+0000 7f909b8fc640 10 monclient: monmap: 2023-10-30T20:45:51.355 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:51.355 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:51.355 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:51.356 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:51.356 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:51.356 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:51.356 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:51.356 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:51.357 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:51.357 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:51.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 1 librados: starting msgr at 2023-10-30T20:45:51.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 1 librados: starting objecter 2023-10-30T20:45:51.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 1 Processor -- start 2023-10-30T20:45:51.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 1 -- start start 2023-10-30T20:45:51.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 1 librados: setting wanted keys 2023-10-30T20:45:51.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 1 librados: calling monclient init 2023-10-30T20:45:51.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 10 monclient: init 2023-10-30T20:45:51.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:51.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T20:45:51.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90940ada60 0x7f909413fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:51.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 10 monclient(hunting): picked mon.noname-b con 0x7f90940ada60 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:51.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90940aea90 0x7f90941c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:51.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 10 monclient(hunting): picked mon.noname-a con 0x7f90940aea90 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:51.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90940b12c0 0x7f9094140b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:51.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 10 monclient(hunting): picked mon.noname-c con 0x7f90940b12c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:51.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:51.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90940b3910 con 0x7f90940aea90 2023-10-30T20:45:51.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:51.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90941da7b0 con 0x7f90940ada60 2023-10-30T20:45:51.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:51.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90941da440 con 0x7f90940b12c0 2023-10-30T20:45:51.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:51.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.337+0000 7f909b8fc640 10 monclient(hunting): authenticate will time out at 2874.626486s 2023-10-30T20:45:51.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9098e97640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90940aea90 0x7f90941c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:51.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9099698640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90940ada60 0x7f909413fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:51.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9099e99640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90940b12c0 0x7f9094140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:51.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9099e99640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90940b12c0 0x7f9094140b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41622/0 (socket says 172.21.15.69:41622) 2023-10-30T20:45:51.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9099e99640 1 -- 172.21.15.69:0/629792672 learned_addr learned my addr 172.21.15.69:0/629792672 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:51.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9099e99640 10 monclient(hunting): get_auth_request con 0x7f90940b12c0 auth_method 0 2023-10-30T20:45:51.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9099e99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:51.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9099e99640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:51.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9099e99640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:51.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9099698640 10 monclient(hunting): get_auth_request con 0x7f90940ada60 auth_method 0 2023-10-30T20:45:51.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9099698640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:51.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9099698640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:51.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9099698640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:51.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9098e97640 10 monclient(hunting): get_auth_request con 0x7f90940aea90 auth_method 0 2023-10-30T20:45:51.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9098e97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:51.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9098e97640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:51.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.338+0000 7f9098e97640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:51.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.339+0000 7f9099e99640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:51.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.339+0000 7f9099e99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:51.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.339+0000 7f9099e99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:51.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.339+0000 7f9099698640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:51.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.339+0000 7f9099698640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:51.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.339+0000 7f9099698640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:51.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.339+0000 7f9098e97640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:51.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.339+0000 7f9098e97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:51.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.339+0000 7f9098e97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:51.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.341+0000 7f9099e99640 10 monclient(hunting): handle_auth_done global_id 18260 payload 824 2023-10-30T20:45:51.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.341+0000 7f9099e99640 1 -- 172.21.15.69:0/629792672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90940ada60 msgr2=0x7f909413fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:51.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.341+0000 7f9099e99640 1 --2- 172.21.15.69:0/629792672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90940ada60 0x7f909413fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:51.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.341+0000 7f9099e99640 1 -- 172.21.15.69:0/629792672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90940aea90 msgr2=0x7f90941c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:51.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.341+0000 7f9099e99640 1 --2- 172.21.15.69:0/629792672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90940aea90 0x7f90941c7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:51.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.341+0000 7f9099e99640 10 monclient: _finish_hunting 0 2023-10-30T20:45:51.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.341+0000 7f9099e99640 1 monclient: found mon.noname-c 2023-10-30T20:45:51.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.341+0000 7f9099e99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:51.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.341+0000 7f9099e99640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:45:51.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.341+0000 7f9099e99640 1 -- 172.21.15.69:0/629792672 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9094141040 con 0x7f90940b12c0 2023-10-30T20:45:51.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.344+0000 7f9099e99640 1 --2- 172.21.15.69:0/629792672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90940b12c0 0x7f9094140b00 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7f908c00c7f0 tx=0x7f908c00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:51.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.344+0000 7f9089ffb640 10 client.?.objecter ms_handle_connect 0x7f90940b12c0 2023-10-30T20:45:51.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.344+0000 7f9089ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f9089ffb640 1 -- 172.21.15.69:0/629792672 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c003b70 con 0x7f90940b12c0 2023-10-30T20:45:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f9089ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f9089ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:45:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f9089ffb640 10 monclient: dump: 2023-10-30T20:45:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:51.371 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:51.371 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.371 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.371 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:51.372 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:51.372 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:51.372 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:51.372 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:51.372 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:51.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f9089ffb640 10 monclient: _finish_auth 0 2023-10-30T20:45:51.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f9089ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:45:51.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f9089ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:51.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f9089ffb640 1 -- 172.21.15.69:0/629792672 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908c00e040 con 0x7f90940b12c0 2023-10-30T20:45:51.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f9089ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:51.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f9089ffb640 1 -- 172.21.15.69:0/629792672 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c00a7f0 con 0x7f90940b12c0 2023-10-30T20:45:51.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f909b8fc640 5 monclient: authenticate success, global_id 18260 2023-10-30T20:45:51.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f9089ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:51.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f9089ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:45:51.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f9089ffb640 10 monclient: dump: 2023-10-30T20:45:51.375 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:51.375 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:51.375 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.376 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.376 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:51.376 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:51.376 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:51.376 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:51.377 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:51.377 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:51.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f909b8fc640 10 monclient: _renew_subs 2023-10-30T20:45:51.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f909b8fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:45:51.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f909b8fc640 1 -- 172.21.15.69:0/629792672 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90941417b0 con 0x7f90940b12c0 2023-10-30T20:45:51.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f908b7fe640 4 set_mon_vals no callback set 2023-10-30T20:45:51.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f909b8fc640 10 client.18260.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:51.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f909b8fc640 10 monclient: _renew_subs 2023-10-30T20:45:51.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f909b8fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:45:51.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f909b8fc640 1 -- 172.21.15.69:0/629792672 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90941dae20 con 0x7f90940b12c0 2023-10-30T20:45:51.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.347+0000 7f909b8fc640 1 librados: init done 2023-10-30T20:45:51.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.348+0000 7f909b8fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:51.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.348+0000 7f909b8fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:51.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.348+0000 7f909b8fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:45:51.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.348+0000 7f909b8fc640 1 -- 172.21.15.69:0/629792672 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90941da960 con 0x7f90940b12c0 2023-10-30T20:45:51.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.592+0000 7f9089ffb640 1 -- 172.21.15.69:0/629792672 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f908c00a960 con 0x7f90940b12c0 2023-10-30T20:45:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.592+0000 7f9089ffb640 1 --2- 172.21.15.69:0/629792672 >> v2:172.21.15.69:6800/1876307353 conn(0x7f907806c810 0x7f907806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.593+0000 7f9099698640 1 --2- 172.21.15.69:0/629792672 >> v2:172.21.15.69:6800/1876307353 conn(0x7f907806c810 0x7f907806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:51.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.593+0000 7f9099698640 10 monclient: get_auth_request con 0x7f907806c810 auth_method 0 2023-10-30T20:45:51.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.593+0000 7f9099698640 1 --2- 172.21.15.69:0/629792672 >> v2:172.21.15.69:6800/1876307353 conn(0x7f907806c810 0x7f907806ecd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9090000f60 tx=0x7f9090009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:51.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.593+0000 7f9089ffb640 10 client.18260.objecter ms_handle_connect 0x7f907806c810 2023-10-30T20:45:51.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.604+0000 7f9089ffb640 1 -- 172.21.15.69:0/629792672 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(903..903 src has 451..903) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f908c058490 con 0x7f90940b12c0 2023-10-30T20:45:51.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.604+0000 7f9089ffb640 10 client.18260.objecter ms_dispatch 0x7f9094000b90 osd_map(903..903 src has 451..903) v4 2023-10-30T20:45:51.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.604+0000 7f9089ffb640 3 client.18260.objecter handle_osd_map got epochs [903,903] > 0 2023-10-30T20:45:51.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.604+0000 7f9089ffb640 3 client.18260.objecter handle_osd_map decoding full epoch 903 2023-10-30T20:45:51.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.605+0000 7f9089ffb640 20 client.18260.objecter dump_active .. 0 homeless 2023-10-30T20:45:51.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.605+0000 7f9089ffb640 1 -- 172.21.15.69:0/629792672 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f908c05c530 con 0x7f90940b12c0 2023-10-30T20:45:51.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.605+0000 7f9089ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:51.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.605+0000 7f9089ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:45:51.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.793+0000 7f909b8fc640 1 -- 172.21.15.69:0/629792672 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90940b0440 con 0x7f907806c810 2023-10-30T20:45:51.801 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.800+0000 7f9089ffb640 1 -- 172.21.15.69:0/629792672 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151627 (secure 0 0 0) 0x7f9090001520 con 0x7f907806c810 2023-10-30T20:45:51.803 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:45:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-10-30T20:45:51.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.802+0000 7f909b8fc640 10 librados: watch_flush enter 2023-10-30T20:45:51.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.802+0000 7f909b8fc640 10 librados: watch_flush exit 2023-10-30T20:45:51.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 20 client.18260.objecter shutdown clearing up homeless session... 2023-10-30T20:45:51.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 10 client.18260.objecter successfully canceled tick 2023-10-30T20:45:51.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 1 -- 172.21.15.69:0/629792672 >> v2:172.21.15.69:6800/1876307353 conn(0x7f907806c810 msgr2=0x7f907806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:51.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 1 --2- 172.21.15.69:0/629792672 >> v2:172.21.15.69:6800/1876307353 conn(0x7f907806c810 0x7f907806ecd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9090000f60 tx=0x7f9090009040 comp rx=0 tx=0).stop 2023-10-30T20:45:51.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 10 monclient: shutdown 2023-10-30T20:45:51.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:51.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 1 -- 172.21.15.69:0/629792672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90940b12c0 msgr2=0x7f9094140b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:51.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 1 --2- 172.21.15.69:0/629792672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90940b12c0 0x7f9094140b00 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7f908c00c7f0 tx=0x7f908c00ccb0 comp rx=0 tx=0).stop 2023-10-30T20:45:51.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 1 -- 172.21.15.69:0/629792672 shutdown_connections 2023-10-30T20:45:51.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 1 --2- 172.21.15.69:0/629792672 >> v2:172.21.15.69:6800/1876307353 conn(0x7f907806c810 0x7f907806ecd0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:51.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 1 --2- 172.21.15.69:0/629792672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90940b12c0 0x7f9094140b00 unknown :-1 s=CLOSED pgs=3199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:51.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 1 --2- 172.21.15.69:0/629792672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90940aea90 0x7f90941c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:51.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 1 --2- 172.21.15.69:0/629792672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90940ada60 0x7f909413fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:51.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 1 -- 172.21.15.69:0/629792672 >> 172.21.15.69:0/629792672 conn(0x7f90940b7120 msgr2=0x7f90940b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:51.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 1 -- 172.21.15.69:0/629792672 shutdown_connections 2023-10-30T20:45:51.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 1 -- 172.21.15.69:0/629792672 wait complete. 2023-10-30T20:45:51.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.803+0000 7f909b8fc640 1 librados: shutdown 2023-10-30T20:45:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1566,"stamp":"2023-10-30T20:45:50.707267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":170,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":545504,"kb_used_data":51748,"kb_used_omap":12,"kb_used_meta":493619,"kb_avail":754429216,"statfs":{"total":773094113280,"available":772535517184,"internally_reserved":0,"allocated":52989952,"data_stored":25970806,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12745,"internal_metadata":505466423},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":402,"apply_latency_ms":402,"commit_latency_ns":402000000,"apply_latency_ns":402000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.204988"},"pg_stats":[{"pgid":"10.16","version":"0'0","reported_seq":845,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.917971+0000","last_change":"2023-10-30T20:25:23.971023+0000","last_active":"2023-10-30T20:45:46.917971+0000","last_peered":"2023-10-30T20:45:46.917971+0000","last_clean":"2023-10-30T20:45:46.917971+0000","last_became_active":"2023-10-30T20:25:23.962078+0000","last_became_peered":"2023-10-30T20:25:23.962078+0000","last_unstale":"2023-10-30T20:45:46.917971+0000","last_undegraded":"2023-10-30T20:45:46.917971+0000","last_fullsized":"2023-10-30T20:45:46.917971+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:31.404506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":869,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.942454+0000","last_change":"2023-10-30T20:25:24.870906+0000","last_active":"2023-10-30T20:45:46.942454+0000","last_peered":"2023-10-30T20:45:46.942454+0000","last_clean":"2023-10-30T20:45:46.942454+0000","last_became_active":"2023-10-30T20:25:24.861824+0000","last_became_peered":"2023-10-30T20:25:24.861824+0000","last_unstale":"2023-10-30T20:45:46.942454+0000","last_undegraded":"2023-10-30T20:45:46.942454+0000","last_fullsized":"2023-10-30T20:45:46.942454+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:40.886429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":847,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T20:45:44.933736+0000","last_change":"2023-10-30T20:25:36.960712+0000","last_active":"2023-10-30T20:45:44.933736+0000","last_peered":"2023-10-30T20:45:44.933736+0000","last_clean":"2023-10-30T20:45:44.933736+0000","last_became_active":"2023-10-30T20:25:36.951760+0000","last_became_peered":"2023-10-30T20:25:36.951760+0000","last_unstale":"2023-10-30T20:45:44.933736+0000","last_undegraded":"2023-10-30T20:45:44.933736+0000","last_fullsized":"2023-10-30T20:45:44.933736+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:17:29.613581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":866,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.937918+0000","last_change":"2023-10-30T20:25:23.364080+0000","last_active":"2023-10-30T20:45:46.937918+0000","last_peered":"2023-10-30T20:45:46.937918+0000","last_clean":"2023-10-30T20:45:46.937918+0000","last_became_active":"2023-10-30T20:25:23.356998+0000","last_became_peered":"2023-10-30T20:25:23.356998+0000","last_unstale":"2023-10-30T20:45:46.937918+0000","last_undegraded":"2023-10-30T20:45:46.937918+0000","last_fullsized":"2023-10-30T20:45:46.937918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:08:01.028636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.468387+0000","last_change":"2023-10-30T20:25:25.485385+0000","last_active":"2023-10-30T20:45:47.468387+0000","last_peered":"2023-10-30T20:45:47.468387+0000","last_clean":"2023-10-30T20:45:47.468387+0000","last_became_active":"2023-10-30T20:25:25.477142+0000","last_became_peered":"2023-10-30T20:25:25.477142+0000","last_unstale":"2023-10-30T20:45:47.468387+0000","last_undegraded":"2023-10-30T20:45:47.468387+0000","last_fullsized":"2023-10-30T20:45:47.468387+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:26:58.277816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":847,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T20:45:44.925204+0000","last_change":"2023-10-30T20:25:23.846753+0000","last_active":"2023-10-30T20:45:44.925204+0000","last_peered":"2023-10-30T20:45:44.925204+0000","last_clean":"2023-10-30T20:45:44.925204+0000","last_became_active":"2023-10-30T20:25:23.840599+0000","last_became_peered":"2023-10-30T20:25:23.840599+0000","last_unstale":"2023-10-30T20:45:44.925204+0000","last_undegraded":"2023-10-30T20:45:44.925204+0000","last_fullsized":"2023-10-30T20:45:44.925204+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:46.666701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.439563+0000","last_change":"2023-10-30T20:25:28.699156+0000","last_active":"2023-10-30T20:45:47.439563+0000","last_peered":"2023-10-30T20:45:47.439563+0000","last_clean":"2023-10-30T20:45:47.439563+0000","last_became_active":"2023-10-30T20:25:28.691162+0000","last_became_peered":"2023-10-30T20:25:28.691162+0000","last_unstale":"2023-10-30T20:45:47.439563+0000","last_undegraded":"2023-10-30T20:45:47.439563+0000","last_fullsized":"2023-10-30T20:45:47.439563+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:51:37.793199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.449465+0000","last_change":"2023-10-30T20:25:36.297471+0000","last_active":"2023-10-30T20:45:47.449465+0000","last_peered":"2023-10-30T20:45:47.449465+0000","last_clean":"2023-10-30T20:45:47.449465+0000","last_became_active":"2023-10-30T20:25:36.286902+0000","last_became_peered":"2023-10-30T20:25:36.286902+0000","last_unstale":"2023-10-30T20:45:47.449465+0000","last_undegraded":"2023-10-30T20:45:47.449465+0000","last_fullsized":"2023-10-30T20:45:47.449465+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:34.822570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":856,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.586217+0000","last_change":"2023-10-30T20:25:25.388067+0000","last_active":"2023-10-30T20:45:47.586217+0000","last_peered":"2023-10-30T20:45:47.586217+0000","last_clean":"2023-10-30T20:45:47.586217+0000","last_became_active":"2023-10-30T20:25:25.367753+0000","last_became_peered":"2023-10-30T20:25:25.367753+0000","last_unstale":"2023-10-30T20:45:47.586217+0000","last_undegraded":"2023-10-30T20:45:47.586217+0000","last_fullsized":"2023-10-30T20:45:47.586217+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:29:40.135060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":854,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.941210+0000","last_change":"2023-10-30T20:25:33.376337+0000","last_active":"2023-10-30T20:45:46.941210+0000","last_peered":"2023-10-30T20:45:46.941210+0000","last_clean":"2023-10-30T20:45:46.941210+0000","last_became_active":"2023-10-30T20:25:33.362871+0000","last_became_peered":"2023-10-30T20:25:33.362871+0000","last_unstale":"2023-10-30T20:45:46.941210+0000","last_undegraded":"2023-10-30T20:45:46.941210+0000","last_fullsized":"2023-10-30T20:45:46.941210+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:57:41.166559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":855,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.894026+0000","last_change":"2023-10-30T20:25:35.098879+0000","last_active":"2023-10-30T20:45:46.894026+0000","last_peered":"2023-10-30T20:45:46.894026+0000","last_clean":"2023-10-30T20:45:46.894026+0000","last_became_active":"2023-10-30T20:25:35.047116+0000","last_became_peered":"2023-10-30T20:25:35.047116+0000","last_unstale":"2023-10-30T20:45:46.894026+0000","last_undegraded":"2023-10-30T20:45:46.894026+0000","last_fullsized":"2023-10-30T20:45:46.894026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:54:18.479663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":869,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.954888+0000","last_change":"2023-10-30T20:25:45.287217+0000","last_active":"2023-10-30T20:45:46.954888+0000","last_peered":"2023-10-30T20:45:46.954888+0000","last_clean":"2023-10-30T20:45:46.954888+0000","last_became_active":"2023-10-30T20:25:45.271565+0000","last_became_peered":"2023-10-30T20:25:45.271565+0000","last_unstale":"2023-10-30T20:45:46.954888+0000","last_undegraded":"2023-10-30T20:45:46.954888+0000","last_fullsized":"2023-10-30T20:45:46.954888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:35:16.997608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":870,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.939320+0000","last_change":"2023-10-30T20:25:28.837575+0000","last_active":"2023-10-30T20:45:46.939320+0000","last_peered":"2023-10-30T20:45:46.939320+0000","last_clean":"2023-10-30T20:45:46.939320+0000","last_became_active":"2023-10-30T20:25:28.829991+0000","last_became_peered":"2023-10-30T20:25:28.829991+0000","last_unstale":"2023-10-30T20:45:46.939320+0000","last_undegraded":"2023-10-30T20:45:46.939320+0000","last_fullsized":"2023-10-30T20:45:46.939320+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:05:54.663012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.937750+0000","last_change":"2023-10-30T20:24:04.700492+0000","last_active":"2023-10-30T20:45:46.937750+0000","last_peered":"2023-10-30T20:45:46.937750+0000","last_clean":"2023-10-30T20:45:46.937750+0000","last_became_active":"2023-10-30T20:24:04.692769+0000","last_became_peered":"2023-10-30T20:24:04.692769+0000","last_unstale":"2023-10-30T20:45:46.937750+0000","last_undegraded":"2023-10-30T20:45:46.937750+0000","last_fullsized":"2023-10-30T20:45:46.937750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:58.321494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":846,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.919523+0000","last_change":"2023-10-30T20:25:27.300642+0000","last_active":"2023-10-30T20:45:46.919523+0000","last_peered":"2023-10-30T20:45:46.919523+0000","last_clean":"2023-10-30T20:45:46.919523+0000","last_became_active":"2023-10-30T20:25:27.291944+0000","last_became_peered":"2023-10-30T20:25:27.291944+0000","last_unstale":"2023-10-30T20:45:46.919523+0000","last_undegraded":"2023-10-30T20:45:46.919523+0000","last_fullsized":"2023-10-30T20:45:46.919523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:32:41.544931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":872,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.908991+0000","last_change":"2023-10-30T20:24:04.738314+0000","last_active":"2023-10-30T20:45:46.908991+0000","last_peered":"2023-10-30T20:45:46.908991+0000","last_clean":"2023-10-30T20:45:46.908991+0000","last_became_active":"2023-10-30T20:24:04.712006+0000","last_became_peered":"2023-10-30T20:24:04.712006+0000","last_unstale":"2023-10-30T20:45:46.908991+0000","last_undegraded":"2023-10-30T20:45:46.908991+0000","last_fullsized":"2023-10-30T20:45:46.908991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:19:34.828603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":847,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.928609+0000","last_change":"2023-10-30T20:25:39.862114+0000","last_active":"2023-10-30T20:45:46.928609+0000","last_peered":"2023-10-30T20:45:46.928609+0000","last_clean":"2023-10-30T20:45:46.928609+0000","last_became_active":"2023-10-30T20:25:39.852846+0000","last_became_peered":"2023-10-30T20:25:39.852846+0000","last_unstale":"2023-10-30T20:45:46.928609+0000","last_undegraded":"2023-10-30T20:45:46.928609+0000","last_fullsized":"2023-10-30T20:45:46.928609+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:00:12.439335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.951807+0000","last_change":"2023-10-30T20:24:04.315228+0000","last_active":"2023-10-30T20:45:46.951807+0000","last_peered":"2023-10-30T20:45:46.951807+0000","last_clean":"2023-10-30T20:45:46.951807+0000","last_became_active":"2023-10-30T20:24:04.310065+0000","last_became_peered":"2023-10-30T20:24:04.310065+0000","last_unstale":"2023-10-30T20:45:46.951807+0000","last_undegraded":"2023-10-30T20:45:46.951807+0000","last_fullsized":"2023-10-30T20:45:46.951807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:09:06.876832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":856,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.895586+0000","last_change":"2023-10-30T20:25:42.330893+0000","last_active":"2023-10-30T20:45:46.895586+0000","last_peered":"2023-10-30T20:45:46.895586+0000","last_clean":"2023-10-30T20:45:46.895586+0000","last_became_active":"2023-10-30T20:25:42.322505+0000","last_became_peered":"2023-10-30T20:25:42.322505+0000","last_unstale":"2023-10-30T20:45:46.895586+0000","last_undegraded":"2023-10-30T20:45:46.895586+0000","last_fullsized":"2023-10-30T20:45:46.895586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:04.210219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":860,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.609161+0000","last_change":"2023-10-30T20:24:04.552803+0000","last_active":"2023-10-30T20:45:47.609161+0000","last_peered":"2023-10-30T20:45:47.609161+0000","last_clean":"2023-10-30T20:45:47.609161+0000","last_became_active":"2023-10-30T20:24:04.525673+0000","last_became_peered":"2023-10-30T20:24:04.525673+0000","last_unstale":"2023-10-30T20:45:47.609161+0000","last_undegraded":"2023-10-30T20:45:47.609161+0000","last_fullsized":"2023-10-30T20:45:47.609161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:18.627320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":856,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.906498+0000","last_change":"2023-10-30T20:25:33.256701+0000","last_active":"2023-10-30T20:45:46.906498+0000","last_peered":"2023-10-30T20:45:46.906498+0000","last_clean":"2023-10-30T20:45:46.906498+0000","last_became_active":"2023-10-30T20:25:33.242618+0000","last_became_peered":"2023-10-30T20:25:33.242618+0000","last_unstale":"2023-10-30T20:45:46.906498+0000","last_undegraded":"2023-10-30T20:45:46.906498+0000","last_fullsized":"2023-10-30T20:45:46.906498+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:56.435103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.910532+0000","last_change":"2023-10-30T20:24:04.525394+0000","last_active":"2023-10-30T20:45:46.910532+0000","last_peered":"2023-10-30T20:45:46.910532+0000","last_clean":"2023-10-30T20:45:46.910532+0000","last_became_active":"2023-10-30T20:24:04.512341+0000","last_became_peered":"2023-10-30T20:24:04.512341+0000","last_unstale":"2023-10-30T20:45:46.910532+0000","last_undegraded":"2023-10-30T20:45:46.910532+0000","last_fullsized":"2023-10-30T20:45:46.910532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:00:36.886515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":847,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.889757+0000","last_change":"2023-10-30T20:25:25.171687+0000","last_active":"2023-10-30T20:45:46.889757+0000","last_peered":"2023-10-30T20:45:46.889757+0000","last_clean":"2023-10-30T20:45:46.889757+0000","last_became_active":"2023-10-30T20:25:25.151444+0000","last_became_peered":"2023-10-30T20:25:25.151444+0000","last_unstale":"2023-10-30T20:45:46.889757+0000","last_undegraded":"2023-10-30T20:45:46.889757+0000","last_fullsized":"2023-10-30T20:45:46.889757+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:25:32.523488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":872,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-10-30T20:45:48.444468+0000","last_change":"2023-10-30T20:24:05.504482+0000","last_active":"2023-10-30T20:45:48.444468+0000","last_peered":"2023-10-30T20:45:48.444468+0000","last_clean":"2023-10-30T20:45:48.444468+0000","last_became_active":"2023-10-30T20:24:05.475394+0000","last_became_peered":"2023-10-30T20:24:05.475394+0000","last_unstale":"2023-10-30T20:45:48.444468+0000","last_undegraded":"2023-10-30T20:45:48.444468+0000","last_fullsized":"2023-10-30T20:45:48.444468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:21:02.335096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":855,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.530050+0000","last_change":"2023-10-30T20:25:23.490954+0000","last_active":"2023-10-30T20:45:47.530050+0000","last_peered":"2023-10-30T20:45:47.530050+0000","last_clean":"2023-10-30T20:45:47.530050+0000","last_became_active":"2023-10-30T20:25:23.473157+0000","last_became_peered":"2023-10-30T20:25:23.473157+0000","last_unstale":"2023-10-30T20:45:47.530050+0000","last_undegraded":"2023-10-30T20:45:47.530050+0000","last_fullsized":"2023-10-30T20:45:47.530050+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:47:35.168516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.896471+0000","last_change":"2023-10-30T20:24:04.284202+0000","last_active":"2023-10-30T20:45:46.896471+0000","last_peered":"2023-10-30T20:45:46.896471+0000","last_clean":"2023-10-30T20:45:46.896471+0000","last_became_active":"2023-10-30T20:24:04.254051+0000","last_became_peered":"2023-10-30T20:24:04.254051+0000","last_unstale":"2023-10-30T20:45:46.896471+0000","last_undegraded":"2023-10-30T20:45:46.896471+0000","last_fullsized":"2023-10-30T20:45:46.896471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:14:06.705421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":878,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.465358+0000","last_change":"2023-10-30T20:25:36.268959+0000","last_active":"2023-10-30T20:45:47.465358+0000","last_peered":"2023-10-30T20:45:47.465358+0000","last_clean":"2023-10-30T20:45:47.465358+0000","last_became_active":"2023-10-30T20:25:36.260481+0000","last_became_peered":"2023-10-30T20:25:36.260481+0000","last_unstale":"2023-10-30T20:45:47.465358+0000","last_undegraded":"2023-10-30T20:45:47.465358+0000","last_fullsized":"2023-10-30T20:45:47.465358+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:06:50.808567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.917896+0000","last_change":"2023-10-30T20:24:04.676966+0000","last_active":"2023-10-30T20:45:46.917896+0000","last_peered":"2023-10-30T20:45:46.917896+0000","last_clean":"2023-10-30T20:45:46.917896+0000","last_became_active":"2023-10-30T20:24:04.671790+0000","last_became_peered":"2023-10-30T20:24:04.671790+0000","last_unstale":"2023-10-30T20:45:46.917896+0000","last_undegraded":"2023-10-30T20:45:46.917896+0000","last_fullsized":"2023-10-30T20:45:46.917896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:56.600896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":989,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.441313+0000","last_change":"2023-10-30T20:23:57.127569+0000","last_active":"2023-10-30T20:45:47.441313+0000","last_peered":"2023-10-30T20:45:47.441313+0000","last_clean":"2023-10-30T20:45:47.441313+0000","last_became_active":"2023-10-30T20:23:57.098872+0000","last_became_peered":"2023-10-30T20:23:57.098872+0000","last_unstale":"2023-10-30T20:45:47.441313+0000","last_undegraded":"2023-10-30T20:45:47.441313+0000","last_fullsized":"2023-10-30T20:45:47.441313+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:23:54.534733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:23:54.534733+0000","last_clean_scrub_stamp":"2023-10-30T20:23:54.534733+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:26:29.039781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"45'7","reported_seq":860,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.892575+0000","last_change":"2023-10-30T20:25:25.378575+0000","last_active":"2023-10-30T20:45:46.892575+0000","last_peered":"2023-10-30T20:45:46.892575+0000","last_clean":"2023-10-30T20:45:46.892575+0000","last_became_active":"2023-10-30T20:25:25.362636+0000","last_became_peered":"2023-10-30T20:25:25.362636+0000","last_unstale":"2023-10-30T20:45:46.892575+0000","last_undegraded":"2023-10-30T20:45:46.892575+0000","last_fullsized":"2023-10-30T20:45:46.892575+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:34.822570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":857,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.915351+0000","last_change":"2023-10-30T20:25:23.612993+0000","last_active":"2023-10-30T20:45:46.915351+0000","last_peered":"2023-10-30T20:45:46.915351+0000","last_clean":"2023-10-30T20:45:46.915351+0000","last_became_active":"2023-10-30T20:25:23.607599+0000","last_became_peered":"2023-10-30T20:25:23.607599+0000","last_unstale":"2023-10-30T20:45:46.915351+0000","last_undegraded":"2023-10-30T20:45:46.915351+0000","last_fullsized":"2023-10-30T20:45:46.915351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:25:35.997128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":867,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.899233+0000","last_change":"2023-10-30T20:25:28.335295+0000","last_active":"2023-10-30T20:45:46.899233+0000","last_peered":"2023-10-30T20:45:46.899233+0000","last_clean":"2023-10-30T20:45:46.899233+0000","last_became_active":"2023-10-30T20:25:28.326296+0000","last_became_peered":"2023-10-30T20:25:28.326296+0000","last_unstale":"2023-10-30T20:45:46.899233+0000","last_undegraded":"2023-10-30T20:45:46.899233+0000","last_fullsized":"2023-10-30T20:45:46.899233+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:41:04.620931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":855,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.483979+0000","last_change":"2023-10-30T20:25:20.677355+0000","last_active":"2023-10-30T20:45:47.483979+0000","last_peered":"2023-10-30T20:45:47.483979+0000","last_clean":"2023-10-30T20:45:47.483979+0000","last_became_active":"2023-10-30T20:25:20.671965+0000","last_became_peered":"2023-10-30T20:25:20.671965+0000","last_unstale":"2023-10-30T20:45:47.483979+0000","last_undegraded":"2023-10-30T20:45:47.483979+0000","last_fullsized":"2023-10-30T20:45:47.483979+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:55:35.886983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":867,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.936385+0000","last_change":"2023-10-30T20:25:28.987986+0000","last_active":"2023-10-30T20:45:46.936385+0000","last_peered":"2023-10-30T20:45:46.936385+0000","last_clean":"2023-10-30T20:45:46.936385+0000","last_became_active":"2023-10-30T20:25:28.979668+0000","last_became_peered":"2023-10-30T20:25:28.979668+0000","last_unstale":"2023-10-30T20:45:46.936385+0000","last_undegraded":"2023-10-30T20:45:46.936385+0000","last_fullsized":"2023-10-30T20:45:46.936385+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:22:41.686629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":867,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.902220+0000","last_change":"2023-10-30T20:25:23.069394+0000","last_active":"2023-10-30T20:45:46.902220+0000","last_peered":"2023-10-30T20:45:46.902220+0000","last_clean":"2023-10-30T20:45:46.902220+0000","last_became_active":"2023-10-30T20:25:23.058176+0000","last_became_peered":"2023-10-30T20:25:23.058176+0000","last_unstale":"2023-10-30T20:45:46.902220+0000","last_undegraded":"2023-10-30T20:45:46.902220+0000","last_fullsized":"2023-10-30T20:45:46.902220+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:33:56.491589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":870,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.940844+0000","last_change":"2023-10-30T20:25:28.823327+0000","last_active":"2023-10-30T20:45:46.940844+0000","last_peered":"2023-10-30T20:45:46.940844+0000","last_clean":"2023-10-30T20:45:46.940844+0000","last_became_active":"2023-10-30T20:25:28.798013+0000","last_became_peered":"2023-10-30T20:25:28.798013+0000","last_unstale":"2023-10-30T20:45:46.940844+0000","last_undegraded":"2023-10-30T20:45:46.940844+0000","last_fullsized":"2023-10-30T20:45:46.940844+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:35:37.042427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":870,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.931482+0000","last_change":"2023-10-30T20:25:23.723655+0000","last_active":"2023-10-30T20:45:46.931482+0000","last_peered":"2023-10-30T20:45:46.931482+0000","last_clean":"2023-10-30T20:45:46.931482+0000","last_became_active":"2023-10-30T20:25:23.709112+0000","last_became_peered":"2023-10-30T20:25:23.709112+0000","last_unstale":"2023-10-30T20:45:46.931482+0000","last_undegraded":"2023-10-30T20:45:46.931482+0000","last_fullsized":"2023-10-30T20:45:46.931482+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:06:46.124278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":847,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T20:45:44.902141+0000","last_change":"2023-10-30T20:25:28.781780+0000","last_active":"2023-10-30T20:45:44.902141+0000","last_peered":"2023-10-30T20:45:44.902141+0000","last_clean":"2023-10-30T20:45:44.902141+0000","last_became_active":"2023-10-30T20:25:28.762953+0000","last_became_peered":"2023-10-30T20:25:28.762953+0000","last_unstale":"2023-10-30T20:45:44.902141+0000","last_undegraded":"2023-10-30T20:45:44.902141+0000","last_fullsized":"2023-10-30T20:45:44.902141+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:09.371593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":866,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.918142+0000","last_change":"2023-10-30T20:25:21.917941+0000","last_active":"2023-10-30T20:45:46.918142+0000","last_peered":"2023-10-30T20:45:46.918142+0000","last_clean":"2023-10-30T20:45:46.918142+0000","last_became_active":"2023-10-30T20:25:21.911569+0000","last_became_peered":"2023-10-30T20:25:21.911569+0000","last_unstale":"2023-10-30T20:45:46.918142+0000","last_undegraded":"2023-10-30T20:45:46.918142+0000","last_fullsized":"2023-10-30T20:45:46.918142+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:32:26.723949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":873,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.897611+0000","last_change":"2023-10-30T20:29:14.877146+0000","last_active":"2023-10-30T20:45:46.897611+0000","last_peered":"2023-10-30T20:45:46.897611+0000","last_clean":"2023-10-30T20:45:46.897611+0000","last_became_active":"2023-10-30T20:29:14.860294+0000","last_became_peered":"2023-10-30T20:29:14.860294+0000","last_unstale":"2023-10-30T20:45:46.897611+0000","last_undegraded":"2023-10-30T20:45:46.897611+0000","last_fullsized":"2023-10-30T20:45:46.897611+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:31:25.822566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.460543+0000","last_change":"2023-10-30T20:25:29.021159+0000","last_active":"2023-10-30T20:45:47.460543+0000","last_peered":"2023-10-30T20:45:47.460543+0000","last_clean":"2023-10-30T20:45:47.460543+0000","last_became_active":"2023-10-30T20:25:29.004167+0000","last_became_peered":"2023-10-30T20:25:29.004167+0000","last_unstale":"2023-10-30T20:45:47.460543+0000","last_undegraded":"2023-10-30T20:45:47.460543+0000","last_fullsized":"2023-10-30T20:45:47.460543+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T08:00:40.390424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477001,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59904,"kb_used_data":6516,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311936,"statfs":{"total":96636764160,"available":96575422464,"internally_reserved":0,"allocated":6672384,"data_stored":3197595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":25.940000000000001,"5min":18.199999999999999,"15min":16.863},"min":{"1min":5.1040000000000001,"5min":5.0529999999999999,"15min":3.9740000000000002},"max":{"1min":156.071,"5min":222.095,"15min":244.999},"last":8.1519999999999992},{"interface":"front","average":{"1min":32.825000000000003,"5min":20.239000000000001,"15min":18.428000000000001},"min":{"1min":5.1020000000000003,"5min":4.9279999999999999,"15min":4.2539999999999996},"max":{"1min":219.797,"5min":219.797,"15min":219.797},"last":6.5129999999999999}]},{"osd":1,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":25.375,"5min":20.100999999999999,"15min":20.373000000000001},"min":{"1min":9.6869999999999994,"5min":5.6500000000000004,"15min":4.5940000000000003},"max":{"1min":143.87200000000001,"5min":231.85499999999999,"15min":256.815},"last":10.204000000000001},{"interface":"front","average":{"1min":22.506,"5min":19.602,"15min":22.550999999999998},"min":{"1min":7.298,"5min":5.7320000000000002,"15min":5.726},"max":{"1min":122.70399999999999,"5min":237.88300000000001,"15min":424.57999999999998},"last":14.696999999999999}]},{"osd":2,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":34.981999999999999,"5min":24.867999999999999,"15min":24.663},"min":{"1min":9.7789999999999999,"5min":6.1539999999999999,"15min":6.1539999999999999},"max":{"1min":267.81099999999998,"5min":267.81099999999998,"15min":835.96799999999996},"last":10.823},{"interface":"front","average":{"1min":37.021999999999998,"5min":26.030999999999999,"15min":23.010000000000002},"min":{"1min":10.771000000000001,"5min":10.173,"15min":8.2479999999999993},"max":{"1min":244.50399999999999,"5min":247.178,"15min":261.50900000000001},"last":13.718999999999999}]},{"osd":4,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":29.966000000000001,"5min":27.501000000000001,"15min":23.704999999999998},"min":{"1min":6.8390000000000004,"5min":6.625,"15min":5.5190000000000001},"max":{"1min":196.904,"5min":247.904,"15min":247.904},"last":9.1969999999999992},{"interface":"front","average":{"1min":29.170999999999999,"5min":25.465,"15min":24.071000000000002},"min":{"1min":7.9900000000000002,"5min":6.8620000000000001,"15min":6.3840000000000003},"max":{"1min":210.88200000000001,"5min":235.64099999999999,"15min":235.64099999999999},"last":15.352}]},{"osd":5,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":35.137999999999998,"5min":26.385999999999999,"15min":25.082999999999998},"min":{"1min":8.4100000000000001,"5min":6.726,"15min":6.726},"max":{"1min":198.76300000000001,"5min":233.28999999999999,"15min":233.28999999999999},"last":17.306999999999999},{"interface":"front","average":{"1min":46.277000000000001,"5min":31.02,"15min":24.657},"min":{"1min":11.351000000000001,"5min":10.645,"15min":7.883},"max":{"1min":227.28299999999999,"5min":244.23500000000001,"15min":244.23500000000001},"last":18.056999999999999}]},{"osd":6,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":41.406999999999996,"5min":30.888000000000002,"15min":27.672999999999998},"min":{"1min":11.547000000000001,"5min":9.1989999999999998,"15min":9.1989999999999998},"max":{"1min":269.15600000000001,"5min":269.15600000000001,"15min":269.15600000000001},"last":20.324999999999999},{"interface":"front","average":{"1min":32.853999999999999,"5min":24.702999999999999,"15min":22.295999999999999},"min":{"1min":6.093,"5min":6.093,"15min":6.093},"max":{"1min":262.53500000000003,"5min":262.53500000000003,"15min":262.53500000000003},"last":13.135}]},{"osd":7,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":29.765999999999998,"5min":27.036000000000001,"15min":24.699000000000002},"min":{"1min":9.7669999999999995,"5min":8.5570000000000004,"15min":7.3799999999999999},"max":{"1min":222.32400000000001,"5min":264.48000000000002,"15min":264.48000000000002},"last":16.132000000000001},{"interface":"front","average":{"1min":31.981999999999999,"5min":27.591000000000001,"15min":26.734999999999999},"min":{"1min":9.2919999999999998,"5min":8.9700000000000006,"15min":8.9700000000000006},"max":{"1min":233.99700000000001,"5min":245.042,"15min":245.042},"last":18.952000000000002}]}]},{"osd":1,"up_from":16,"seq":68719477001,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64036,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94307804,"statfs":{"total":96636764160,"available":96571191296,"internally_reserved":0,"allocated":6635520,"data_stored":3528420,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.762,"5min":15.228999999999999,"15min":16.553999999999998},"min":{"1min":4.1130000000000004,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":98.356999999999999,"5min":244.65000000000001,"15min":244.65000000000001},"last":4.8230000000000004},{"interface":"front","average":{"1min":11.536,"5min":12.617000000000001,"15min":18.32},"min":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":3.847},"max":{"1min":39.68,"5min":65.853999999999999,"15min":407.38900000000001},"last":9.5150000000000006}]},{"osd":2,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":18.402000000000001,"5min":16.823,"15min":19.931000000000001},"min":{"1min":7.5389999999999997,"5min":7.367,"15min":4.1500000000000004},"max":{"1min":80.688000000000002,"5min":113.649,"15min":297.26499999999999},"last":13.631},{"interface":"front","average":{"1min":23.346,"5min":19.442,"15min":20.483000000000001},"min":{"1min":9.8040000000000003,"5min":8.5690000000000008,"15min":6.5469999999999997},"max":{"1min":148.03999999999999,"5min":148.03999999999999,"15min":167.12100000000001},"last":14.256}]},{"osd":3,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":17.946000000000002,"5min":19.791,"15min":21.096},"min":{"1min":8.8569999999999993,"5min":7.8029999999999999,"15min":6.5990000000000002},"max":{"1min":84.090999999999994,"5min":150.166,"15min":213.971},"last":15.526},{"interface":"front","average":{"1min":19.449000000000002,"5min":22.609000000000002,"15min":24.210999999999999},"min":{"1min":12.093999999999999,"5min":9.8409999999999993,"15min":8.1449999999999996},"max":{"1min":74.275999999999996,"5min":146.15899999999999,"15min":223.49199999999999},"last":16.167000000000002}]},{"osd":4,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":13.090999999999999,"5min":20.369,"15min":21.076000000000001},"min":{"1min":8.1349999999999998,"5min":6.8659999999999997,"15min":5.9809999999999999},"max":{"1min":30.533000000000001,"5min":200.98400000000001,"15min":215.435},"last":10.723000000000001},{"interface":"front","average":{"1min":17.289000000000001,"5min":23.798999999999999,"15min":25.364000000000001},"min":{"1min":12.874000000000001,"5min":11.516999999999999,"15min":8.9939999999999998},"max":{"1min":26.600000000000001,"5min":192.69200000000001,"15min":197.34},"last":17.629999999999999}]},{"osd":5,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":13.349,"5min":19.747,"15min":19.356000000000002},"min":{"1min":7.4020000000000001,"5min":6.4580000000000002,"15min":5.1390000000000002},"max":{"1min":62.798000000000002,"5min":209.35499999999999,"15min":241.798},"last":7.5170000000000003},{"interface":"front","average":{"1min":17.783000000000001,"5min":20.292000000000002,"15min":21.498000000000001},"min":{"1min":9.5739999999999998,"5min":6.6230000000000002,"15min":6.6230000000000002},"max":{"1min":68.882000000000005,"5min":118.65600000000001,"15min":248.124},"last":11.372}]},{"osd":6,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.948,"5min":20.902000000000001,"15min":23.236999999999998},"min":{"1min":10.205,"5min":7.7220000000000004,"15min":6.9880000000000004},"max":{"1min":37.037999999999997,"5min":204.71899999999999,"15min":406.72399999999999},"last":12.73},{"interface":"front","average":{"1min":28.690999999999999,"5min":25.748000000000001,"15min":25.416},"min":{"1min":11.704000000000001,"5min":10.397,"15min":8.4749999999999996},"max":{"1min":179.98099999999999,"5min":179.98099999999999,"15min":179.98099999999999},"last":18.324000000000002}]},{"osd":7,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":14.643000000000001,"5min":18.478999999999999,"15min":23.765000000000001},"min":{"1min":6.0410000000000004,"5min":6.0410000000000004,"15min":6.0110000000000001},"max":{"1min":58.841999999999999,"5min":188.792,"15min":249.88300000000001},"last":8.7400000000000002},{"interface":"front","average":{"1min":19.361000000000001,"5min":21.233000000000001,"15min":25.628},"min":{"1min":12.742000000000001,"5min":8.8559999999999999,"15min":7.1360000000000001},"max":{"1min":59.718000000000004,"5min":134.89400000000001,"15min":275.36700000000002},"last":14.907}]}]},{"osd":0,"up_from":16,"seq":68719477002,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59832,"kb_used_data":6448,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312008,"statfs":{"total":96636764160,"available":96575496192,"internally_reserved":0,"allocated":6602752,"data_stored":3130585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":56,"apply_latency_ms":56,"commit_latency_ns":56000000,"apply_latency_ns":56000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":14.162000000000001,"5min":13.208,"15min":18.428000000000001},"min":{"1min":5.508,"5min":4.1740000000000004,"15min":4.0819999999999999},"max":{"1min":64.745999999999995,"5min":145.857,"15min":341.85300000000001},"last":6.968},{"interface":"front","average":{"1min":15.507,"5min":16.523,"15min":18.965},"min":{"1min":5.9400000000000004,"5min":5.798,"15min":5.1669999999999998},"max":{"1min":71.578000000000003,"5min":196.148,"15min":243.62100000000001},"last":21.934999999999999}]},{"osd":2,"last update":"Mon Oct 30 20:44:59 2023","interfaces":[{"interface":"back","average":{"1min":18.457000000000001,"5min":19.640999999999998,"15min":31.206},"min":{"1min":6.5940000000000003,"5min":4.6449999999999996,"15min":4.6449999999999996},"max":{"1min":68.856999999999999,"5min":199.71100000000001,"15min":580.43100000000004},"last":6.0309999999999997},{"interface":"front","average":{"1min":15.906000000000001,"5min":18.309000000000001,"15min":31.786999999999999},"min":{"1min":6.6920000000000002,"5min":6.6920000000000002,"15min":6.1420000000000003},"max":{"1min":55.250999999999998,"5min":197.125,"15min":1090.171},"last":15.888}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.609,"5min":16.076000000000001,"15min":16.977},"min":{"1min":6.2199999999999998,"5min":5.298,"15min":5.298},"max":{"1min":76.861000000000004,"5min":148.31299999999999,"15min":167.607},"last":8.2200000000000006},{"interface":"front","average":{"1min":30.317,"5min":25.641999999999999,"15min":24.917999999999999},"min":{"1min":7.1719999999999997,"5min":7.1719999999999997,"15min":6.4370000000000003},"max":{"1min":290.19600000000003,"5min":290.19600000000003,"15min":318.245},"last":23.213999999999999}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":24.504000000000001,"5min":20.890000000000001,"15min":20.295000000000002},"min":{"1min":7.234,"5min":7.1059999999999999,"15min":5.8170000000000002},"max":{"1min":116.98399999999999,"5min":174.67699999999999,"15min":178.642},"last":11.069000000000001},{"interface":"front","average":{"1min":30.776,"5min":28.367999999999999,"15min":27.843},"min":{"1min":8.3770000000000007,"5min":8.1010000000000009,"15min":6.2960000000000003},"max":{"1min":153.124,"5min":219.14699999999999,"15min":242.06800000000001},"last":33.905999999999999}]},{"osd":5,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.25,"5min":23.315999999999999,"15min":22.43},"min":{"1min":9.3379999999999992,"5min":8.3480000000000008,"15min":6.8099999999999996},"max":{"1min":68.546999999999997,"5min":170.024,"15min":170.024},"last":26.283999999999999},{"interface":"front","average":{"1min":25.843,"5min":23.294,"15min":22.381},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.8280000000000003},"max":{"1min":138.041,"5min":177.41800000000001,"15min":177.41800000000001},"last":25.364999999999998}]},{"osd":6,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":27.763999999999999,"5min":29.074999999999999,"15min":30.167000000000002},"min":{"1min":10.817,"5min":8.7599999999999998,"15min":8.2590000000000003},"max":{"1min":151.73500000000001,"5min":189.21899999999999,"15min":330.37799999999999},"last":34.863999999999997},{"interface":"front","average":{"1min":19.260000000000002,"5min":20.901,"15min":24.004000000000001},"min":{"1min":6.8410000000000002,"5min":6.8410000000000002,"15min":6.8410000000000002},"max":{"1min":73.170000000000002,"5min":169.447,"15min":201.50299999999999},"last":28.603999999999999}]},{"osd":7,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":20.75,"5min":23.949999999999999,"15min":27.219999999999999},"min":{"1min":11.298,"5min":10.81,"15min":9.6820000000000004},"max":{"1min":70.683000000000007,"5min":175.667,"15min":268.18900000000002},"last":30.404},{"interface":"front","average":{"1min":21.437999999999999,"5min":21.204999999999998,"15min":25.626000000000001},"min":{"1min":8.8689999999999998,"5min":8.8689999999999998,"15min":7.3559999999999999},"max":{"1min":77.927000000000007,"5min":173.441,"15min":228.495},"last":27.532}]}]},{"osd":6,"up_from":15,"seq":64424509705,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59840,"kb_used_data":6444,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312000,"statfs":{"total":96636764160,"available":96575488000,"internally_reserved":0,"allocated":6598656,"data_stored":3155698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":54,"apply_latency_ms":54,"commit_latency_ns":54000000,"apply_latency_ns":54000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:44:48 2023","interfaces":[{"interface":"back","average":{"1min":13.409000000000001,"5min":11.871,"15min":16.204999999999998},"min":{"1min":4.3979999999999997,"5min":4.0179999999999998,"15min":3.7080000000000002},"max":{"1min":73.316000000000003,"5min":73.316000000000003,"15min":158.14599999999999},"last":12.824999999999999},{"interface":"front","average":{"1min":10.07,"5min":12.122,"15min":19.719000000000001},"min":{"1min":5.407,"5min":5.2439999999999998,"15min":4.5499999999999998},"max":{"1min":54.936999999999998,"5min":75.060000000000002,"15min":488.43799999999999},"last":17.391999999999999}]},{"osd":1,"last update":"Mon Oct 30 20:44:48 2023","interfaces":[{"interface":"back","average":{"1min":13.942,"5min":15.932,"15min":20.992999999999999},"min":{"1min":5.4379999999999997,"5min":5.4379999999999997,"15min":4.9199999999999999},"max":{"1min":38.279000000000003,"5min":117.193,"15min":218.78100000000001},"last":6.5860000000000003},{"interface":"front","average":{"1min":15.507,"5min":14.162000000000001,"15min":16.847000000000001},"min":{"1min":5.3230000000000004,"5min":5.1379999999999999,"15min":5.0149999999999997},"max":{"1min":103.065,"5min":128.50200000000001,"15min":155.27199999999999},"last":10.08}]},{"osd":2,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":19.477,"5min":16.728000000000002,"15min":21.635000000000002},"min":{"1min":7.9080000000000004,"5min":6.1870000000000003,"15min":6.1870000000000003},"max":{"1min":67.552999999999997,"5min":122.307,"15min":222.303},"last":10.866},{"interface":"front","average":{"1min":16.923999999999999,"5min":18.561,"15min":26.841000000000001},"min":{"1min":7.9580000000000002,"5min":7.9580000000000002,"15min":6.1820000000000004},"max":{"1min":46.755000000000003,"5min":98.781000000000006,"15min":438.19200000000001},"last":11.613}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.385000000000002,"5min":15.81,"15min":17.850000000000001},"min":{"1min":6.4859999999999998,"5min":6.2050000000000001,"15min":6.0179999999999998},"max":{"1min":136.16800000000001,"5min":136.16800000000001,"15min":217.99000000000001},"last":7.3499999999999996},{"interface":"front","average":{"1min":30.402000000000001,"5min":20.652999999999999,"15min":21.940000000000001},"min":{"1min":6.3890000000000002,"5min":6.3890000000000002,"15min":6.0010000000000003},"max":{"1min":270.25599999999997,"5min":270.25599999999997,"15min":270.25599999999997},"last":15.512}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.241,"5min":21.978000000000002,"15min":23.396000000000001},"min":{"1min":11.669,"5min":8.1539999999999999,"15min":7.5800000000000001},"max":{"1min":69.150000000000006,"5min":168.38800000000001,"15min":190.29599999999999},"last":34.412999999999997},{"interface":"front","average":{"1min":14.085000000000001,"5min":20.285,"15min":22.73},"min":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"max":{"1min":48.207000000000001,"5min":163.41800000000001,"15min":189.59399999999999},"last":9.3260000000000005}]},{"osd":5,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":21.626999999999999,"5min":20.792999999999999,"15min":22.007999999999999},"min":{"1min":9.1430000000000007,"5min":8.9939999999999998,"15min":8.9930000000000003},"max":{"1min":68.114000000000004,"5min":133.73400000000001,"15min":138.38800000000001},"last":24.062000000000001},{"interface":"front","average":{"1min":25.091000000000001,"5min":24.527000000000001,"15min":25.061},"min":{"1min":12.673999999999999,"5min":11.9,"15min":10.571},"max":{"1min":68.138999999999996,"5min":150.35499999999999,"15min":152.40199999999999},"last":19.061}]},{"osd":7,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":17.923999999999999,"5min":18.893999999999998,"15min":23.824999999999999},"min":{"1min":8.8140000000000001,"5min":7.1699999999999999,"15min":6.226},"max":{"1min":64.828999999999994,"5min":98.503,"15min":253.16900000000001},"last":16.713000000000001},{"interface":"front","average":{"1min":19.021000000000001,"5min":21.308,"15min":20.143000000000001},"min":{"1min":7.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"max":{"1min":75.846000000000004,"5min":174.071,"15min":221.28399999999999},"last":12.241}]}]},{"osd":5,"up_from":15,"seq":64424509703,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77920,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293920,"statfs":{"total":96636764160,"available":96556974080,"internally_reserved":0,"allocated":6635520,"data_stored":3528420,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":17.625,"5min":18.785,"15min":16.260999999999999},"min":{"1min":5.2329999999999997,"5min":3.8359999999999999,"15min":3.8359999999999999},"max":{"1min":85.513999999999996,"5min":220.148,"15min":220.148},"last":10.542999999999999},{"interface":"front","average":{"1min":27.861000000000001,"5min":23.675000000000001,"15min":20.949000000000002},"min":{"1min":8.6039999999999992,"5min":5.6059999999999999,"15min":5.6059999999999999},"max":{"1min":145.43000000000001,"5min":177.91,"15min":177.91},"last":14.894}]},{"osd":1,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":20.715,"5min":16.585000000000001,"15min":20.353999999999999},"min":{"1min":6.3040000000000003,"5min":4.3010000000000002,"15min":4.3010000000000002},"max":{"1min":71.460999999999999,"5min":74.081000000000003,"15min":293.75400000000002},"last":9.6920000000000002},{"interface":"front","average":{"1min":20.943999999999999,"5min":16.175000000000001,"15min":18.327999999999999},"min":{"1min":5.46,"5min":4.8949999999999996,"15min":4.8949999999999996},"max":{"1min":80.397000000000006,"5min":80.397000000000006,"15min":210.797},"last":12.271000000000001}]},{"osd":2,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":31.280000000000001,"5min":23.431999999999999,"15min":25.231000000000002},"min":{"1min":6.2480000000000002,"5min":5.0209999999999999,"15min":5.0209999999999999},"max":{"1min":105.449,"5min":143.273,"15min":519.68799999999999},"last":13.766},{"interface":"front","average":{"1min":23.294,"5min":20.007000000000001,"15min":22.029},"min":{"1min":5.6479999999999997,"5min":5.6479999999999997,"15min":5.226},"max":{"1min":82.042000000000002,"5min":154.083,"15min":218.59899999999999},"last":6.0469999999999997}]},{"osd":3,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":41.817,"5min":31.161999999999999,"15min":26.263000000000002},"min":{"1min":9.9900000000000002,"5min":9.9900000000000002,"15min":7.6150000000000002},"max":{"1min":269.52999999999997,"5min":269.52999999999997,"15min":269.52999999999997},"last":16.454999999999998},{"interface":"front","average":{"1min":24.488,"5min":22.629000000000001,"15min":22.704000000000001},"min":{"1min":7.4480000000000004,"5min":6.218,"15min":5.9119999999999999},"max":{"1min":178.78700000000001,"5min":178.78700000000001,"15min":249.797},"last":6.7670000000000003}]},{"osd":4,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":21.725999999999999,"5min":26.437999999999999,"15min":25.25},"min":{"1min":8.1080000000000005,"5min":7.1580000000000004,"15min":6.6100000000000003},"max":{"1min":78.555999999999997,"5min":215.97999999999999,"15min":291.92599999999999},"last":8.1080000000000005},{"interface":"front","average":{"1min":26.097000000000001,"5min":33.600000000000001,"15min":28.585000000000001},"min":{"1min":11.895,"5min":8.0050000000000008,"15min":6.9550000000000001},"max":{"1min":70.290999999999997,"5min":324.31,"15min":324.31},"last":13.007999999999999}]},{"osd":6,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":32.883000000000003,"5min":28.116,"15min":26.379999999999999},"min":{"1min":10.593,"5min":8.6489999999999991,"15min":7.2199999999999998},"max":{"1min":162.964,"5min":211.255,"15min":251.37700000000001},"last":8.7439999999999998},{"interface":"front","average":{"1min":38.536000000000001,"5min":33.917999999999999,"15min":28.524000000000001},"min":{"1min":9.3640000000000008,"5min":9.2349999999999994,"15min":7.1619999999999999},"max":{"1min":160.42599999999999,"5min":333.29700000000003,"15min":333.29700000000003},"last":17.245000000000001}]},{"osd":7,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":36.252000000000002,"5min":33.210000000000001,"15min":27.071999999999999},"min":{"1min":11.131,"5min":10.68,"15min":6.1769999999999996},"max":{"1min":116.877,"5min":258.81,"15min":264.52999999999997},"last":15.683999999999999},{"interface":"front","average":{"1min":26.748000000000001,"5min":22.239000000000001,"15min":25.803999999999998},"min":{"1min":7.4580000000000002,"5min":5.8410000000000002,"15min":5.8410000000000002},"max":{"1min":90.570999999999998,"5min":154.86799999999999,"15min":255.625},"last":11.167999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542408,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59852,"kb_used_data":6456,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311988,"statfs":{"total":96636764160,"available":96575475712,"internally_reserved":0,"allocated":6610944,"data_stored":3136155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":9.2010000000000005,"5min":13.369999999999999,"15min":14.285},"min":{"1min":4.9580000000000002,"5min":4.0499999999999998,"15min":3.5920000000000001},"max":{"1min":18.266999999999999,"5min":106.79300000000001,"15min":187.85499999999999},"last":18.699999999999999},{"interface":"front","average":{"1min":8.7729999999999997,"5min":12.766999999999999,"15min":13.725},"min":{"1min":5.0279999999999996,"5min":4.1020000000000003,"15min":4.1020000000000003},"max":{"1min":15.933,"5min":117.645,"15min":124.342},"last":7.9589999999999996}]},{"osd":1,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":13.706,"5min":18.475999999999999,"15min":19.167999999999999},"min":{"1min":8.1969999999999992,"5min":5.8979999999999997,"15min":5.7409999999999997},"max":{"1min":33.970999999999997,"5min":103.907,"15min":226.63399999999999},"last":19.806000000000001},{"interface":"front","average":{"1min":11.929,"5min":17.565999999999999,"15min":19.605},"min":{"1min":6.5439999999999996,"5min":6.351,"15min":5.859},"max":{"1min":42.719000000000001,"5min":135.41200000000001,"15min":187.24100000000001},"last":11.147}]},{"osd":2,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":17.675000000000001,"5min":18.248999999999999,"15min":20.965},"min":{"1min":7.2869999999999999,"5min":6.2549999999999999,"15min":6.2549999999999999},"max":{"1min":43.064999999999998,"5min":79.168000000000006,"15min":200.285},"last":13.004},{"interface":"front","average":{"1min":26.209,"5min":21.07,"15min":21.149999999999999},"min":{"1min":6.0679999999999996,"5min":5.5090000000000003,"15min":5.5090000000000003},"max":{"1min":211.315,"5min":211.315,"15min":241.15700000000001},"last":5.4720000000000004}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":25.942,"5min":21.164000000000001,"15min":22.437999999999999},"min":{"1min":10.516,"5min":8.1509999999999998,"15min":6.6849999999999996},"max":{"1min":205.821,"5min":205.821,"15min":262.43900000000002},"last":14.073},{"interface":"front","average":{"1min":26.780000000000001,"5min":22.042999999999999,"15min":23.443999999999999},"min":{"1min":7.5650000000000004,"5min":6.7460000000000004,"15min":5.9950000000000001},"max":{"1min":225.29900000000001,"5min":225.29900000000001,"15min":330.15300000000002},"last":10.228}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.153,"5min":21.227,"15min":22.390000000000001},"min":{"1min":9.9459999999999997,"5min":7.218,"15min":7.218},"max":{"1min":22.241,"5min":136.494,"15min":213.072},"last":15.015000000000001},{"interface":"front","average":{"1min":17.73,"5min":22.32,"15min":24.960999999999999},"min":{"1min":8.9540000000000006,"5min":8.9540000000000006,"15min":8.5530000000000008},"max":{"1min":28.087,"5min":104.538,"15min":239.76900000000001},"last":17.381}]},{"osd":5,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.374000000000001,"5min":18.962,"15min":22.965},"min":{"1min":10.206,"5min":8.2889999999999997,"15min":7.258},"max":{"1min":27.122,"5min":110.45099999999999,"15min":275.08100000000002},"last":12.374000000000001},{"interface":"front","average":{"1min":13.379,"5min":15.702999999999999,"15min":18.018000000000001},"min":{"1min":5.3410000000000002,"5min":5.3410000000000002,"15min":5.3410000000000002},"max":{"1min":40.390000000000001,"5min":94.302999999999997,"15min":129.935},"last":7.3300000000000001}]},{"osd":6,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":18.283000000000001,"5min":21.891999999999999,"15min":25.209},"min":{"1min":11.725,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":27.867999999999999,"5min":126.001,"15min":263.41399999999999},"last":16.436},{"interface":"front","average":{"1min":17.962,"5min":19.896000000000001,"15min":23.355},"min":{"1min":11.305999999999999,"5min":8.0229999999999997,"15min":8.0229999999999997},"max":{"1min":56.701000000000001,"5min":111.691,"15min":257.75900000000001},"last":20.581}]}]},{"osd":2,"up_from":14,"seq":60129542408,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82048,"kb_used_data":6448,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289792,"statfs":{"total":96636764160,"available":96552747008,"internally_reserved":0,"allocated":6602752,"data_stored":3130585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":65,"apply_latency_ms":65,"commit_latency_ns":65000000,"apply_latency_ns":65000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":19.824999999999999,"5min":15.673,"15min":19.306999999999999},"min":{"1min":4.6959999999999997,"5min":4.3170000000000002,"15min":4.3170000000000002},"max":{"1min":211.80500000000001,"5min":211.80500000000001,"15min":389.31400000000002},"last":269.779},{"interface":"front","average":{"1min":18.757999999999999,"5min":20.440999999999999,"15min":19.539000000000001},"min":{"1min":5.9269999999999996,"5min":5.3040000000000003,"15min":4.4589999999999996},"max":{"1min":213.93600000000001,"5min":295.87,"15min":295.87},"last":124.283}]},{"osd":1,"last update":"Mon Oct 30 20:45:48 2023","interfaces":[{"interface":"back","average":{"1min":14.757,"5min":16.274999999999999,"15min":22.295999999999999},"min":{"1min":6.4720000000000004,"5min":5.5060000000000002,"15min":5.2649999999999997},"max":{"1min":36.689,"5min":74.575000000000003,"15min":245.84999999999999},"last":9.8390000000000004},{"interface":"front","average":{"1min":17.843,"5min":15.342000000000001,"15min":20.946999999999999},"min":{"1min":7.9459999999999997,"5min":5.8399999999999999,"15min":5.3330000000000002},"max":{"1min":66.947999999999993,"5min":72.662999999999997,"15min":317.20699999999999},"last":13.996}]},{"osd":3,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.094000000000001,"5min":17.843,"15min":22.158000000000001},"min":{"1min":6.6040000000000001,"5min":6.1459999999999999,"15min":5.2619999999999996},"max":{"1min":58.442999999999998,"5min":253.02799999999999,"15min":311.03100000000001},"last":8.6120000000000001},{"interface":"front","average":{"1min":20.954000000000001,"5min":21.210999999999999,"15min":28.140999999999998},"min":{"1min":10.403,"5min":7.4969999999999999,"15min":6.2039999999999997},"max":{"1min":86.935000000000002,"5min":145.066,"15min":703},"last":10.464}]},{"osd":4,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":14.718,"5min":18.241,"15min":26.344999999999999},"min":{"1min":7.4029999999999996,"5min":6.4809999999999999,"15min":6.3460000000000001},"max":{"1min":43.838999999999999,"5min":123.06399999999999,"15min":709.85900000000004},"last":12.739000000000001},{"interface":"front","average":{"1min":16.712,"5min":19.233000000000001,"15min":22.349},"min":{"1min":6.2359999999999998,"5min":6.2359999999999998,"15min":6.2359999999999998},"max":{"1min":65.897000000000006,"5min":118.82599999999999,"15min":182.947},"last":28.228000000000002}]},{"osd":5,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":18.765000000000001,"5min":18.805,"15min":29.754000000000001},"min":{"1min":8.3539999999999992,"5min":7.2279999999999998,"15min":6.8460000000000001},"max":{"1min":63.963000000000001,"5min":116.352,"15min":874.22799999999995},"last":31.942},{"interface":"front","average":{"1min":17.289999999999999,"5min":18.369,"15min":24.251999999999999},"min":{"1min":6.9500000000000002,"5min":6.9500000000000002,"15min":6.9500000000000002},"max":{"1min":41.155000000000001,"5min":73.218999999999994,"15min":222.76400000000001},"last":26.239999999999998}]},{"osd":6,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":15.76,"5min":23.898,"15min":25.614999999999998},"min":{"1min":8.5389999999999997,"5min":8.5389999999999997,"15min":7.3049999999999997},"max":{"1min":37.296999999999997,"5min":196.654,"15min":238.84800000000001},"last":28.916},{"interface":"front","average":{"1min":18.132000000000001,"5min":23.280000000000001,"15min":29.577000000000002},"min":{"1min":8.3770000000000007,"5min":8.0670000000000002,"15min":7.577},"max":{"1min":37.128999999999998,"5min":161.30799999999999,"15min":266.48599999999999},"last":31.02}]},{"osd":7,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":19.760000000000002,"5min":20.834,"15min":26.890000000000001},"min":{"1min":9.7300000000000004,"5min":9.7300000000000004,"15min":9.7300000000000004},"max":{"1min":39.747,"5min":133.429,"15min":354.904},"last":30.41},{"interface":"front","average":{"1min":19.949999999999999,"5min":21.204000000000001,"15min":29.919},"min":{"1min":9.1600000000000001,"5min":7.3440000000000003,"15min":7.3440000000000003},"max":{"1min":75.688999999999993,"5min":153.803,"15min":865.25800000000004},"last":13.436}]}]},{"osd":4,"up_from":13,"seq":55834575114,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82072,"kb_used_data":6476,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289768,"statfs":{"total":96636764160,"available":96552722432,"internally_reserved":0,"allocated":6631424,"data_stored":3163348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":59,"apply_latency_ms":59,"commit_latency_ns":59000000,"apply_latency_ns":59000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":12.099,"5min":10.295,"15min":15.339},"min":{"1min":4.5940000000000003,"5min":3.8839999999999999,"15min":3.839},"max":{"1min":73.688000000000002,"5min":118.02500000000001,"15min":224.95099999999999},"last":8.1910000000000007},{"interface":"front","average":{"1min":14.191000000000001,"5min":11.253,"15min":15.319000000000001},"min":{"1min":4.407,"5min":4.407,"15min":4.0229999999999997},"max":{"1min":70.055999999999997,"5min":70.055999999999997,"15min":227.71600000000001},"last":7.0209999999999999}]},{"osd":1,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":19.260999999999999,"5min":15.923,"15min":20.707999999999998},"min":{"1min":6.3550000000000004,"5min":4.7960000000000003,"15min":4.7960000000000003},"max":{"1min":92.763000000000005,"5min":118.747,"15min":383.57499999999999},"last":7.4740000000000002},{"interface":"front","average":{"1min":20.491,"5min":19.440000000000001,"15min":22.885999999999999},"min":{"1min":8.3000000000000007,"5min":7.8970000000000002,"15min":7.5},"max":{"1min":59.844999999999999,"5min":110.129,"15min":228.34999999999999},"last":10.853999999999999}]},{"osd":2,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":17.518000000000001,"5min":13.864000000000001,"15min":21.329000000000001},"min":{"1min":8.2390000000000008,"5min":4.6420000000000003,"15min":4.6420000000000003},"max":{"1min":54.417999999999999,"5min":82.471999999999994,"15min":347.30799999999999},"last":15.742000000000001},{"interface":"front","average":{"1min":23.317,"5min":16.451000000000001,"15min":21.233000000000001},"min":{"1min":6.1369999999999996,"5min":6.1369999999999996,"15min":6.0090000000000003},"max":{"1min":121.45999999999999,"5min":121.45999999999999,"15min":183.03100000000001},"last":11.478}]},{"osd":3,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":16.106000000000002,"5min":17.736000000000001,"15min":24.536000000000001},"min":{"1min":6.4779999999999998,"5min":6.4779999999999998,"15min":6.2930000000000001},"max":{"1min":70.343999999999994,"5min":174.733,"15min":270.19400000000002},"last":12.321999999999999},{"interface":"front","average":{"1min":18.103999999999999,"5min":17.949999999999999,"15min":24.756},"min":{"1min":6.0220000000000002,"5min":6.0220000000000002,"15min":6.0220000000000002},"max":{"1min":63.222000000000001,"5min":177.87,"15min":339.79700000000003},"last":9.3510000000000009}]},{"osd":5,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":20.681000000000001,"5min":20.640999999999998,"15min":24.623999999999999},"min":{"1min":6.5839999999999996,"5min":6.5839999999999996,"15min":6.5839999999999996},"max":{"1min":158.541,"5min":158.541,"15min":253.80199999999999},"last":16.981999999999999},{"interface":"front","average":{"1min":17.707000000000001,"5min":16.873000000000001,"15min":22.225000000000001},"min":{"1min":6.1029999999999998,"5min":6.1029999999999998,"15min":6.1029999999999998},"max":{"1min":147.54400000000001,"5min":147.54400000000001,"15min":271.572},"last":13.77}]},{"osd":6,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":26.273,"5min":18.638999999999999,"15min":24.821999999999999},"min":{"1min":7.0999999999999996,"5min":6.4210000000000003,"15min":6.4210000000000003},"max":{"1min":245.447,"5min":245.447,"15min":254.44399999999999},"last":15.035},{"interface":"front","average":{"1min":18.928999999999998,"5min":19.975000000000001,"15min":28.715},"min":{"1min":10.25,"5min":9.0419999999999998,"15min":8.1639999999999997},"max":{"1min":72.599000000000004,"5min":74.463999999999999,"15min":291.62599999999998},"last":14.419}]},{"osd":7,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":14.513,"5min":16.260000000000002,"15min":22.474},"min":{"1min":8.1890000000000001,"5min":8.1890000000000001,"15min":6.2389999999999999},"max":{"1min":43.911999999999999,"5min":76.325000000000003,"15min":262.82100000000003},"last":17.632999999999999},{"interface":"front","average":{"1min":19.515000000000001,"5min":19.870999999999999,"15min":25.632000000000001},"min":{"1min":9.8350000000000009,"5min":9.8350000000000009,"15min":9.8350000000000009},"max":{"1min":61.091000000000001,"5min":61.091000000000001,"15min":244.37799999999999},"last":18.791}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T20:45:51.818 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T20:45:51.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.917+0000 7f8359383640 10 monclient: get_monmap_and_config 2023-10-30T20:45:51.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.917+0000 7f8359383640 10 monclient: build_initial_monmap 2023-10-30T20:45:51.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.917+0000 7f8359383640 10 monclient: monmap: 2023-10-30T20:45:51.919 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:51.919 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:51.919 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:51.918281+0000 2023-10-30T20:45:51.919 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:51.918281+0000 2023-10-30T20:45:51.920 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:51.920 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:51.920 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:51.920 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:51.920 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:51.921 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.917+0000 7f8359383640 1 Processor -- start 2023-10-30T20:45:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.917+0000 7f8359383640 1 -- start start 2023-10-30T20:45:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.917+0000 7f8359383640 10 monclient: init 2023-10-30T20:45:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.918+0000 7f8359383640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.918+0000 7f8359383640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-10-30T20:45:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.918+0000 7f8359383640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83540ad660 0x7f835406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.918+0000 7f8359383640 10 monclient(hunting): picked mon.noname-b con 0x7f83540ad660 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.918+0000 7f8359383640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83540ada40 addr v1:172.21.15.69:6789/0 2023-10-30T20:45:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.918+0000 7f8359383640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8354141af0 0x7f83540717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.918+0000 7f8359383640 10 monclient(hunting): picked mon.noname-a con 0x7f8354141af0 addr v2:172.21.15.69:3300/0 2023-10-30T20:45:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.918+0000 7f8359383640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8354070cb0 con 0x7f83540ada40 2023-10-30T20:45:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.918+0000 7f8359383640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.918+0000 7f8359383640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8354067b90 con 0x7f8354141af0 2023-10-30T20:45:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.918+0000 7f8359383640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.918+0000 7f8359383640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8354071300 con 0x7f83540ad660 2023-10-30T20:45:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.918+0000 7f8359383640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.918+0000 7f8359383640 10 monclient(hunting): authenticate will time out at 2875.207435s 2023-10-30T20:45:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.919+0000 7f8352ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83540ad660 0x7f835406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.919+0000 7f83537fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8354141af0 0x7f83540717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.919+0000 7f83537fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8354141af0 0x7f83540717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50586/0 (socket says 172.21.15.69:50586) 2023-10-30T20:45:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.919+0000 7f83537fe640 1 -- 172.21.15.69:0/2823655252 learned_addr learned my addr 172.21.15.69:0/2823655252 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.919+0000 7f8352ffd640 10 monclient(hunting): get_auth_request con 0x7f83540ad660 auth_method 0 2023-10-30T20:45:51.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.919+0000 7f8352ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:51.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.919+0000 7f8352ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:51.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.919+0000 7f8352ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:51.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.919+0000 7f83537fe640 10 monclient(hunting): get_auth_request con 0x7f8354141af0 auth_method 0 2023-10-30T20:45:51.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.919+0000 7f83537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:51.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.919+0000 7f83537fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:51.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.919+0000 7f83537fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:51.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.920+0000 7f8352ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:51.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.920+0000 7f8352ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:51.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.920+0000 7f8352ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:51.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.920+0000 7f83537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.920+0000 7f83537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.920+0000 7f83537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.921+0000 7f8352ffd640 10 monclient(hunting): handle_auth_done global_id 18263 payload 274 2023-10-30T20:45:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.921+0000 7f8352ffd640 1 -- 172.21.15.69:0/2823655252 >> v2:172.21.15.69:3300/0 conn(0x7f8354141af0 msgr2=0x7f83540717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.921+0000 7f8352ffd640 1 --2- 172.21.15.69:0/2823655252 >> v2:172.21.15.69:3300/0 conn(0x7f8354141af0 0x7f83540717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.921+0000 7f8352ffd640 1 -- 172.21.15.69:0/2823655252 >> v1:172.21.15.69:6789/0 conn(0x7f83540ada40 legacy=0x7f835413bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.921+0000 7f8352ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:45:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.921+0000 7f8352ffd640 1 monclient: found mon.noname-b 2023-10-30T20:45:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.921+0000 7f8352ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:51.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.921+0000 7f8352ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:45:51.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.921+0000 7f8352ffd640 1 -- 172.21.15.69:0/2823655252 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8354070ee0 con 0x7f83540ad660 2023-10-30T20:45:51.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.923+0000 7f8352ffd640 1 --2- 172.21.15.69:0/2823655252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83540ad660 0x7f835406ec00 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 crypto rx=0x7f8344009d90 tx=0x7f834402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=525faa5243d0ce71 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:51.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8351ffb640 1 -- 172.21.15.69:0/2823655252 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402cbc0 con 0x7f83540ad660 2023-10-30T20:45:51.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:51.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8351ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:51.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8351ffb640 10 monclient: dump: 2023-10-30T20:45:51.936 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:51.936 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:51.936 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.936 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.937 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:51.937 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:51.937 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:51.937 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:51.937 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:51.937 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:51.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8351ffb640 10 monclient: _finish_auth 0 2023-10-30T20:45:51.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8351ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:45:51.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8351ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:51.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8351ffb640 1 -- 172.21.15.69:0/2823655252 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834402cd30 con 0x7f83540ad660 2023-10-30T20:45:51.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8351ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:51.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8351ffb640 1 -- 172.21.15.69:0/2823655252 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344034660 con 0x7f83540ad660 2023-10-30T20:45:51.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8359383640 5 monclient: authenticate success, global_id 18263 2023-10-30T20:45:51.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8359383640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:51.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8359383640 4 set_mon_vals no callback set 2023-10-30T20:45:51.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:51.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8351ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:45:51.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.927+0000 7f8351ffb640 10 monclient: dump: 2023-10-30T20:45:51.940 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:51.940 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:51.940 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.940 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.941 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:51.941 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:51.941 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:51.941 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:51.941 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:51.942 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:51.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.928+0000 7f8359383640 10 monclient: shutdown 2023-10-30T20:45:51.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.928+0000 7f8359383640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:51.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.928+0000 7f8359383640 1 -- 172.21.15.69:0/2823655252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83540ad660 msgr2=0x7f835406ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:51.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.928+0000 7f8359383640 1 --2- 172.21.15.69:0/2823655252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83540ad660 0x7f835406ec00 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 crypto rx=0x7f8344009d90 tx=0x7f834402be10 comp rx=0 tx=0).stop 2023-10-30T20:45:51.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.928+0000 7f8359383640 1 -- 172.21.15.69:0/2823655252 shutdown_connections 2023-10-30T20:45:51.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.928+0000 7f8359383640 1 --2- 172.21.15.69:0/2823655252 >> v2:172.21.15.69:3300/0 conn(0x7f8354141af0 0x7f83540717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:51.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.928+0000 7f8359383640 1 --2- 172.21.15.69:0/2823655252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83540ad660 0x7f835406ec00 unknown :-1 s=CLOSED pgs=3200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:51.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.928+0000 7f8359383640 1 -- 172.21.15.69:0/2823655252 >> 172.21.15.69:0/2823655252 conn(0x7f83540afb80 msgr2=0x7f83540aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:51.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.928+0000 7f8359383640 1 -- 172.21.15.69:0/2823655252 shutdown_connections 2023-10-30T20:45:51.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.928+0000 7f8359383640 1 -- 172.21.15.69:0/2823655252 wait complete. 2023-10-30T20:45:51.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.929+0000 7f8359383640 10 monclient: build_initial_monmap 2023-10-30T20:45:51.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.929+0000 7f8359383640 10 monclient: monmap: 2023-10-30T20:45:51.944 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:51.944 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:51.944 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:51.945 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:51.945 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:51.945 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:51.945 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:51.945 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:51.946 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:51.946 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:51.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.929+0000 7f8359383640 1 librados: starting msgr at 2023-10-30T20:45:51.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.929+0000 7f8359383640 1 librados: starting objecter 2023-10-30T20:45:51.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.929+0000 7f8359383640 1 Processor -- start 2023-10-30T20:45:51.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.929+0000 7f8359383640 1 -- start start 2023-10-30T20:45:51.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.929+0000 7f8359383640 1 librados: setting wanted keys 2023-10-30T20:45:51.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.929+0000 7f8359383640 1 librados: calling monclient init 2023-10-30T20:45:51.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.929+0000 7f8359383640 10 monclient: init 2023-10-30T20:45:51.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:51.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T20:45:51.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83540ad660 0x7f83541d2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:51.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 10 monclient(hunting): picked mon.noname-c con 0x7f83540ad660 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:51.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8354141af0 0x7f835413e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:51.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 10 monclient(hunting): picked mon.noname-b con 0x7f8354141af0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:51.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83541cf910 0x7f83541cfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:51.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 10 monclient(hunting): picked mon.noname-a con 0x7f83541cf910 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835406f660 con 0x7f83541cf910 2023-10-30T20:45:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835413dd60 con 0x7f8354141af0 2023-10-30T20:45:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:51.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835413e040 con 0x7f83540ad660 2023-10-30T20:45:51.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:51.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.930+0000 7f8359383640 10 monclient(hunting): authenticate will time out at 2875.219408s 2023-10-30T20:45:51.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.931+0000 7f8352ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83540ad660 0x7f83541d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:51.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.931+0000 7f83537fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83541cf910 0x7f83541cfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:51.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.931+0000 7f8352ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83540ad660 0x7f83541d2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41640/0 (socket says 172.21.15.69:41640) 2023-10-30T20:45:51.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.931+0000 7f8352ffd640 1 -- 172.21.15.69:0/881145109 learned_addr learned my addr 172.21.15.69:0/881145109 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:51.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.931+0000 7f8352ffd640 10 monclient(hunting): get_auth_request con 0x7f83540ad660 auth_method 0 2023-10-30T20:45:51.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.931+0000 7f8352ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:51.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.931+0000 7f8352ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.931+0000 7f8352ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.931+0000 7f83537fe640 10 monclient(hunting): get_auth_request con 0x7f83541cf910 auth_method 0 2023-10-30T20:45:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.931+0000 7f83537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.931+0000 7f83537fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.931+0000 7f83537fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.932+0000 7f8352ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:51.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.932+0000 7f8352ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:51.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.932+0000 7f8352ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:51.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.932+0000 7f83537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:51.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.932+0000 7f83537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:51.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.932+0000 7f83537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:51.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.936+0000 7f8352ffd640 10 monclient(hunting): handle_auth_done global_id 18266 payload 824 2023-10-30T20:45:51.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.936+0000 7f8352ffd640 1 -- 172.21.15.69:0/881145109 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8354141af0 msgr2=0x7f835413e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:51.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.936+0000 7f8352ffd640 1 --2- 172.21.15.69:0/881145109 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8354141af0 0x7f835413e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:51.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.936+0000 7f8352ffd640 1 -- 172.21.15.69:0/881145109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83541cf910 msgr2=0x7f83541cfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:51.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.936+0000 7f8352ffd640 1 --2- 172.21.15.69:0/881145109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83541cf910 0x7f83541cfdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:51.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.936+0000 7f8352ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:45:51.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.936+0000 7f8352ffd640 1 monclient: found mon.noname-c 2023-10-30T20:45:51.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.936+0000 7f8352ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:51.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.936+0000 7f8352ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:45:51.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.936+0000 7f8352ffd640 1 -- 172.21.15.69:0/881145109 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83541d2bd0 con 0x7f83540ad660 2023-10-30T20:45:51.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.938+0000 7f8352ffd640 1 --2- 172.21.15.69:0/881145109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83540ad660 0x7f83541d2690 secure :-1 s=READY pgs=3201 cs=0 l=1 rev1=1 crypto rx=0x7f8344009eb0 tx=0x7f8344004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:51.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.938+0000 7f833b7fe640 10 client.?.objecter ms_handle_connect 0x7f83540ad660 2023-10-30T20:45:51.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.938+0000 7f833b7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:51.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f833b7fe640 1 -- 172.21.15.69:0/881145109 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403e610 con 0x7f83540ad660 2023-10-30T20:45:51.958 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f833b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:51.958 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f833b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:45:51.958 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f833b7fe640 10 monclient: dump: 2023-10-30T20:45:51.958 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:51.958 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:51.958 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.959 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.959 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:51.959 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:51.959 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:51.959 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:51.960 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:51.960 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:51.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f833b7fe640 10 monclient: _finish_auth 0 2023-10-30T20:45:51.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f833b7fe640 10 monclient: _check_auth_tickets 2023-10-30T20:45:51.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f833b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:51.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f833b7fe640 1 -- 172.21.15.69:0/881145109 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834403e7a0 con 0x7f83540ad660 2023-10-30T20:45:51.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f833b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:51.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f833b7fe640 1 -- 172.21.15.69:0/881145109 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403eb60 con 0x7f83540ad660 2023-10-30T20:45:51.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f8359383640 5 monclient: authenticate success, global_id 18266 2023-10-30T20:45:51.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f833b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:51.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f833b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:45:51.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f833b7fe640 10 monclient: dump: 2023-10-30T20:45:51.962 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:51.962 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:51.962 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.962 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:51.963 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:51.963 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:51.963 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:51.963 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:51.963 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:51.964 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:51.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f8359383640 10 monclient: _renew_subs 2023-10-30T20:45:51.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f83517fa640 4 set_mon_vals no callback set 2023-10-30T20:45:51.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f8359383640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:45:51.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f8359383640 1 -- 172.21.15.69:0/881145109 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835413ea10 con 0x7f83540ad660 2023-10-30T20:45:51.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f8359383640 10 client.18266.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:51.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f8359383640 10 monclient: _renew_subs 2023-10-30T20:45:51.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f8359383640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:45:51.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f8359383640 1 -- 172.21.15.69:0/881145109 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835413ffe0 con 0x7f83540ad660 2023-10-30T20:45:51.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.942+0000 7f8359383640 1 librados: init done 2023-10-30T20:45:51.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.943+0000 7f8359383640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:51.966 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.943+0000 7f8359383640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:51.966 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.943+0000 7f8359383640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:45:51.966 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.943+0000 7f8359383640 1 -- 172.21.15.69:0/881145109 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f835413c240 con 0x7f83540ad660 2023-10-30T20:45:51.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.961+0000 7f833b7fe640 1 -- 172.21.15.69:0/881145109 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f834403b050 con 0x7f83540ad660 2023-10-30T20:45:51.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.962+0000 7f833b7fe640 1 --2- 172.21.15.69:0/881145109 >> v2:172.21.15.69:6800/1876307353 conn(0x7f834006c810 0x7f834006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:51.967 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.962+0000 7f83527fc640 1 --2- 172.21.15.69:0/881145109 >> v2:172.21.15.69:6800/1876307353 conn(0x7f834006c810 0x7f834006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:51.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.962+0000 7f83527fc640 10 monclient: get_auth_request con 0x7f834006c810 auth_method 0 2023-10-30T20:45:51.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.962+0000 7f833b7fe640 10 client.18266.objecter ms_handle_connect 0x7f834006c810 2023-10-30T20:45:51.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.962+0000 7f83527fc640 1 --2- 172.21.15.69:0/881145109 >> v2:172.21.15.69:6800/1876307353 conn(0x7f834006c810 0x7f834006ecd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8348000f60 tx=0x7f83480079d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:51.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.965+0000 7f833b7fe640 1 -- 172.21.15.69:0/881145109 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(903..903 src has 451..903) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f834407a8c0 con 0x7f83540ad660 2023-10-30T20:45:51.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.965+0000 7f833b7fe640 10 client.18266.objecter ms_dispatch 0x7f8354000b90 osd_map(903..903 src has 451..903) v4 2023-10-30T20:45:51.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.965+0000 7f833b7fe640 3 client.18266.objecter handle_osd_map got epochs [903,903] > 0 2023-10-30T20:45:51.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.965+0000 7f833b7fe640 3 client.18266.objecter handle_osd_map decoding full epoch 903 2023-10-30T20:45:51.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:51.965+0000 7f833b7fe640 20 client.18266.objecter dump_active .. 0 homeless 2023-10-30T20:45:52.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.201+0000 7f833b7fe640 1 -- 172.21.15.69:0/881145109 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f834407e960 con 0x7f83540ad660 2023-10-30T20:45:52.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.201+0000 7f833b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:52.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.201+0000 7f833b7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:45:52.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.362+0000 7f8359383640 1 -- 172.21.15.69:0/881145109 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8354070ee0 con 0x7f834006c810 2023-10-30T20:45:52.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.367+0000 7f833b7fe640 1 -- 172.21.15.69:0/881145109 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151627 (secure 0 0 0) 0x7f8354070ee0 con 0x7f834006c810 2023-10-30T20:45:52.370 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:45:52.371 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-10-30T20:45:52.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 10 librados: watch_flush enter 2023-10-30T20:45:52.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 10 librados: watch_flush exit 2023-10-30T20:45:52.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 20 client.18266.objecter shutdown clearing up homeless session... 2023-10-30T20:45:52.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 10 client.18266.objecter successfully canceled tick 2023-10-30T20:45:52.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 1 -- 172.21.15.69:0/881145109 >> v2:172.21.15.69:6800/1876307353 conn(0x7f834006c810 msgr2=0x7f834006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:52.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 1 --2- 172.21.15.69:0/881145109 >> v2:172.21.15.69:6800/1876307353 conn(0x7f834006c810 0x7f834006ecd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8348000f60 tx=0x7f83480079d0 comp rx=0 tx=0).stop 2023-10-30T20:45:52.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 10 monclient: shutdown 2023-10-30T20:45:52.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:52.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 1 -- 172.21.15.69:0/881145109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83540ad660 msgr2=0x7f83541d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:52.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 1 --2- 172.21.15.69:0/881145109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83540ad660 0x7f83541d2690 secure :-1 s=READY pgs=3201 cs=0 l=1 rev1=1 crypto rx=0x7f8344009eb0 tx=0x7f8344004a40 comp rx=0 tx=0).stop 2023-10-30T20:45:52.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 1 -- 172.21.15.69:0/881145109 shutdown_connections 2023-10-30T20:45:52.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 1 --2- 172.21.15.69:0/881145109 >> v2:172.21.15.69:6800/1876307353 conn(0x7f834006c810 0x7f834006ecd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:52.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 1 --2- 172.21.15.69:0/881145109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83541cf910 0x7f83541cfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:52.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 1 --2- 172.21.15.69:0/881145109 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8354141af0 0x7f835413e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:52.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 1 --2- 172.21.15.69:0/881145109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83540ad660 0x7f83541d2690 unknown :-1 s=CLOSED pgs=3201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:52.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 1 -- 172.21.15.69:0/881145109 >> 172.21.15.69:0/881145109 conn(0x7f83540afb80 msgr2=0x7f83540aff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 1 -- 172.21.15.69:0/881145109 shutdown_connections 2023-10-30T20:45:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.369+0000 7f8359383640 1 -- 172.21.15.69:0/881145109 wait complete. 2023-10-30T20:45:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.370+0000 7f8359383640 1 librados: shutdown 2023-10-30T20:45:52.382 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1566,"stamp":"2023-10-30T20:45:50.707267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":170,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":545504,"kb_used_data":51748,"kb_used_omap":12,"kb_used_meta":493619,"kb_avail":754429216,"statfs":{"total":773094113280,"available":772535517184,"internally_reserved":0,"allocated":52989952,"data_stored":25970806,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12745,"internal_metadata":505466423},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":402,"apply_latency_ms":402,"commit_latency_ns":402000000,"apply_latency_ns":402000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.204988"},"pg_stats":[{"pgid":"10.16","version":"0'0","reported_seq":845,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.917971+0000","last_change":"2023-10-30T20:25:23.971023+0000","last_active":"2023-10-30T20:45:46.917971+0000","last_peered":"2023-10-30T20:45:46.917971+0000","last_clean":"2023-10-30T20:45:46.917971+0000","last_became_active":"2023-10-30T20:25:23.962078+0000","last_became_peered":"2023-10-30T20:25:23.962078+0000","last_unstale":"2023-10-30T20:45:46.917971+0000","last_undegraded":"2023-10-30T20:45:46.917971+0000","last_fullsized":"2023-10-30T20:45:46.917971+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:31.404506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":869,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.942454+0000","last_change":"2023-10-30T20:25:24.870906+0000","last_active":"2023-10-30T20:45:46.942454+0000","last_peered":"2023-10-30T20:45:46.942454+0000","last_clean":"2023-10-30T20:45:46.942454+0000","last_became_active":"2023-10-30T20:25:24.861824+0000","last_became_peered":"2023-10-30T20:25:24.861824+0000","last_unstale":"2023-10-30T20:45:46.942454+0000","last_undegraded":"2023-10-30T20:45:46.942454+0000","last_fullsized":"2023-10-30T20:45:46.942454+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:40.886429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":847,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T20:45:44.933736+0000","last_change":"2023-10-30T20:25:36.960712+0000","last_active":"2023-10-30T20:45:44.933736+0000","last_peered":"2023-10-30T20:45:44.933736+0000","last_clean":"2023-10-30T20:45:44.933736+0000","last_became_active":"2023-10-30T20:25:36.951760+0000","last_became_peered":"2023-10-30T20:25:36.951760+0000","last_unstale":"2023-10-30T20:45:44.933736+0000","last_undegraded":"2023-10-30T20:45:44.933736+0000","last_fullsized":"2023-10-30T20:45:44.933736+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:17:29.613581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":866,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.937918+0000","last_change":"2023-10-30T20:25:23.364080+0000","last_active":"2023-10-30T20:45:46.937918+0000","last_peered":"2023-10-30T20:45:46.937918+0000","last_clean":"2023-10-30T20:45:46.937918+0000","last_became_active":"2023-10-30T20:25:23.356998+0000","last_became_peered":"2023-10-30T20:25:23.356998+0000","last_unstale":"2023-10-30T20:45:46.937918+0000","last_undegraded":"2023-10-30T20:45:46.937918+0000","last_fullsized":"2023-10-30T20:45:46.937918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:08:01.028636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.468387+0000","last_change":"2023-10-30T20:25:25.485385+0000","last_active":"2023-10-30T20:45:47.468387+0000","last_peered":"2023-10-30T20:45:47.468387+0000","last_clean":"2023-10-30T20:45:47.468387+0000","last_became_active":"2023-10-30T20:25:25.477142+0000","last_became_peered":"2023-10-30T20:25:25.477142+0000","last_unstale":"2023-10-30T20:45:47.468387+0000","last_undegraded":"2023-10-30T20:45:47.468387+0000","last_fullsized":"2023-10-30T20:45:47.468387+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:26:58.277816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":847,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T20:45:44.925204+0000","last_change":"2023-10-30T20:25:23.846753+0000","last_active":"2023-10-30T20:45:44.925204+0000","last_peered":"2023-10-30T20:45:44.925204+0000","last_clean":"2023-10-30T20:45:44.925204+0000","last_became_active":"2023-10-30T20:25:23.840599+0000","last_became_peered":"2023-10-30T20:25:23.840599+0000","last_unstale":"2023-10-30T20:45:44.925204+0000","last_undegraded":"2023-10-30T20:45:44.925204+0000","last_fullsized":"2023-10-30T20:45:44.925204+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:46.666701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.439563+0000","last_change":"2023-10-30T20:25:28.699156+0000","last_active":"2023-10-30T20:45:47.439563+0000","last_peered":"2023-10-30T20:45:47.439563+0000","last_clean":"2023-10-30T20:45:47.439563+0000","last_became_active":"2023-10-30T20:25:28.691162+0000","last_became_peered":"2023-10-30T20:25:28.691162+0000","last_unstale":"2023-10-30T20:45:47.439563+0000","last_undegraded":"2023-10-30T20:45:47.439563+0000","last_fullsized":"2023-10-30T20:45:47.439563+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:51:37.793199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.449465+0000","last_change":"2023-10-30T20:25:36.297471+0000","last_active":"2023-10-30T20:45:47.449465+0000","last_peered":"2023-10-30T20:45:47.449465+0000","last_clean":"2023-10-30T20:45:47.449465+0000","last_became_active":"2023-10-30T20:25:36.286902+0000","last_became_peered":"2023-10-30T20:25:36.286902+0000","last_unstale":"2023-10-30T20:45:47.449465+0000","last_undegraded":"2023-10-30T20:45:47.449465+0000","last_fullsized":"2023-10-30T20:45:47.449465+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:34.822570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":856,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.586217+0000","last_change":"2023-10-30T20:25:25.388067+0000","last_active":"2023-10-30T20:45:47.586217+0000","last_peered":"2023-10-30T20:45:47.586217+0000","last_clean":"2023-10-30T20:45:47.586217+0000","last_became_active":"2023-10-30T20:25:25.367753+0000","last_became_peered":"2023-10-30T20:25:25.367753+0000","last_unstale":"2023-10-30T20:45:47.586217+0000","last_undegraded":"2023-10-30T20:45:47.586217+0000","last_fullsized":"2023-10-30T20:45:47.586217+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:29:40.135060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":854,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.941210+0000","last_change":"2023-10-30T20:25:33.376337+0000","last_active":"2023-10-30T20:45:46.941210+0000","last_peered":"2023-10-30T20:45:46.941210+0000","last_clean":"2023-10-30T20:45:46.941210+0000","last_became_active":"2023-10-30T20:25:33.362871+0000","last_became_peered":"2023-10-30T20:25:33.362871+0000","last_unstale":"2023-10-30T20:45:46.941210+0000","last_undegraded":"2023-10-30T20:45:46.941210+0000","last_fullsized":"2023-10-30T20:45:46.941210+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:57:41.166559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":855,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.894026+0000","last_change":"2023-10-30T20:25:35.098879+0000","last_active":"2023-10-30T20:45:46.894026+0000","last_peered":"2023-10-30T20:45:46.894026+0000","last_clean":"2023-10-30T20:45:46.894026+0000","last_became_active":"2023-10-30T20:25:35.047116+0000","last_became_peered":"2023-10-30T20:25:35.047116+0000","last_unstale":"2023-10-30T20:45:46.894026+0000","last_undegraded":"2023-10-30T20:45:46.894026+0000","last_fullsized":"2023-10-30T20:45:46.894026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:54:18.479663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":869,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.954888+0000","last_change":"2023-10-30T20:25:45.287217+0000","last_active":"2023-10-30T20:45:46.954888+0000","last_peered":"2023-10-30T20:45:46.954888+0000","last_clean":"2023-10-30T20:45:46.954888+0000","last_became_active":"2023-10-30T20:25:45.271565+0000","last_became_peered":"2023-10-30T20:25:45.271565+0000","last_unstale":"2023-10-30T20:45:46.954888+0000","last_undegraded":"2023-10-30T20:45:46.954888+0000","last_fullsized":"2023-10-30T20:45:46.954888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:35:16.997608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":870,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.939320+0000","last_change":"2023-10-30T20:25:28.837575+0000","last_active":"2023-10-30T20:45:46.939320+0000","last_peered":"2023-10-30T20:45:46.939320+0000","last_clean":"2023-10-30T20:45:46.939320+0000","last_became_active":"2023-10-30T20:25:28.829991+0000","last_became_peered":"2023-10-30T20:25:28.829991+0000","last_unstale":"2023-10-30T20:45:46.939320+0000","last_undegraded":"2023-10-30T20:45:46.939320+0000","last_fullsized":"2023-10-30T20:45:46.939320+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:05:54.663012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.937750+0000","last_change":"2023-10-30T20:24:04.700492+0000","last_active":"2023-10-30T20:45:46.937750+0000","last_peered":"2023-10-30T20:45:46.937750+0000","last_clean":"2023-10-30T20:45:46.937750+0000","last_became_active":"2023-10-30T20:24:04.692769+0000","last_became_peered":"2023-10-30T20:24:04.692769+0000","last_unstale":"2023-10-30T20:45:46.937750+0000","last_undegraded":"2023-10-30T20:45:46.937750+0000","last_fullsized":"2023-10-30T20:45:46.937750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:58.321494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":846,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.919523+0000","last_change":"2023-10-30T20:25:27.300642+0000","last_active":"2023-10-30T20:45:46.919523+0000","last_peered":"2023-10-30T20:45:46.919523+0000","last_clean":"2023-10-30T20:45:46.919523+0000","last_became_active":"2023-10-30T20:25:27.291944+0000","last_became_peered":"2023-10-30T20:25:27.291944+0000","last_unstale":"2023-10-30T20:45:46.919523+0000","last_undegraded":"2023-10-30T20:45:46.919523+0000","last_fullsized":"2023-10-30T20:45:46.919523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:32:41.544931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":872,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.908991+0000","last_change":"2023-10-30T20:24:04.738314+0000","last_active":"2023-10-30T20:45:46.908991+0000","last_peered":"2023-10-30T20:45:46.908991+0000","last_clean":"2023-10-30T20:45:46.908991+0000","last_became_active":"2023-10-30T20:24:04.712006+0000","last_became_peered":"2023-10-30T20:24:04.712006+0000","last_unstale":"2023-10-30T20:45:46.908991+0000","last_undegraded":"2023-10-30T20:45:46.908991+0000","last_fullsized":"2023-10-30T20:45:46.908991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:19:34.828603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":847,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.928609+0000","last_change":"2023-10-30T20:25:39.862114+0000","last_active":"2023-10-30T20:45:46.928609+0000","last_peered":"2023-10-30T20:45:46.928609+0000","last_clean":"2023-10-30T20:45:46.928609+0000","last_became_active":"2023-10-30T20:25:39.852846+0000","last_became_peered":"2023-10-30T20:25:39.852846+0000","last_unstale":"2023-10-30T20:45:46.928609+0000","last_undegraded":"2023-10-30T20:45:46.928609+0000","last_fullsized":"2023-10-30T20:45:46.928609+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:00:12.439335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.951807+0000","last_change":"2023-10-30T20:24:04.315228+0000","last_active":"2023-10-30T20:45:46.951807+0000","last_peered":"2023-10-30T20:45:46.951807+0000","last_clean":"2023-10-30T20:45:46.951807+0000","last_became_active":"2023-10-30T20:24:04.310065+0000","last_became_peered":"2023-10-30T20:24:04.310065+0000","last_unstale":"2023-10-30T20:45:46.951807+0000","last_undegraded":"2023-10-30T20:45:46.951807+0000","last_fullsized":"2023-10-30T20:45:46.951807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:09:06.876832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":856,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.895586+0000","last_change":"2023-10-30T20:25:42.330893+0000","last_active":"2023-10-30T20:45:46.895586+0000","last_peered":"2023-10-30T20:45:46.895586+0000","last_clean":"2023-10-30T20:45:46.895586+0000","last_became_active":"2023-10-30T20:25:42.322505+0000","last_became_peered":"2023-10-30T20:25:42.322505+0000","last_unstale":"2023-10-30T20:45:46.895586+0000","last_undegraded":"2023-10-30T20:45:46.895586+0000","last_fullsized":"2023-10-30T20:45:46.895586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:04.210219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":860,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.609161+0000","last_change":"2023-10-30T20:24:04.552803+0000","last_active":"2023-10-30T20:45:47.609161+0000","last_peered":"2023-10-30T20:45:47.609161+0000","last_clean":"2023-10-30T20:45:47.609161+0000","last_became_active":"2023-10-30T20:24:04.525673+0000","last_became_peered":"2023-10-30T20:24:04.525673+0000","last_unstale":"2023-10-30T20:45:47.609161+0000","last_undegraded":"2023-10-30T20:45:47.609161+0000","last_fullsized":"2023-10-30T20:45:47.609161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:18.627320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":856,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.906498+0000","last_change":"2023-10-30T20:25:33.256701+0000","last_active":"2023-10-30T20:45:46.906498+0000","last_peered":"2023-10-30T20:45:46.906498+0000","last_clean":"2023-10-30T20:45:46.906498+0000","last_became_active":"2023-10-30T20:25:33.242618+0000","last_became_peered":"2023-10-30T20:25:33.242618+0000","last_unstale":"2023-10-30T20:45:46.906498+0000","last_undegraded":"2023-10-30T20:45:46.906498+0000","last_fullsized":"2023-10-30T20:45:46.906498+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:56.435103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.910532+0000","last_change":"2023-10-30T20:24:04.525394+0000","last_active":"2023-10-30T20:45:46.910532+0000","last_peered":"2023-10-30T20:45:46.910532+0000","last_clean":"2023-10-30T20:45:46.910532+0000","last_became_active":"2023-10-30T20:24:04.512341+0000","last_became_peered":"2023-10-30T20:24:04.512341+0000","last_unstale":"2023-10-30T20:45:46.910532+0000","last_undegraded":"2023-10-30T20:45:46.910532+0000","last_fullsized":"2023-10-30T20:45:46.910532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:00:36.886515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":847,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.889757+0000","last_change":"2023-10-30T20:25:25.171687+0000","last_active":"2023-10-30T20:45:46.889757+0000","last_peered":"2023-10-30T20:45:46.889757+0000","last_clean":"2023-10-30T20:45:46.889757+0000","last_became_active":"2023-10-30T20:25:25.151444+0000","last_became_peered":"2023-10-30T20:25:25.151444+0000","last_unstale":"2023-10-30T20:45:46.889757+0000","last_undegraded":"2023-10-30T20:45:46.889757+0000","last_fullsized":"2023-10-30T20:45:46.889757+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:25:32.523488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":872,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-10-30T20:45:48.444468+0000","last_change":"2023-10-30T20:24:05.504482+0000","last_active":"2023-10-30T20:45:48.444468+0000","last_peered":"2023-10-30T20:45:48.444468+0000","last_clean":"2023-10-30T20:45:48.444468+0000","last_became_active":"2023-10-30T20:24:05.475394+0000","last_became_peered":"2023-10-30T20:24:05.475394+0000","last_unstale":"2023-10-30T20:45:48.444468+0000","last_undegraded":"2023-10-30T20:45:48.444468+0000","last_fullsized":"2023-10-30T20:45:48.444468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:21:02.335096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":855,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.530050+0000","last_change":"2023-10-30T20:25:23.490954+0000","last_active":"2023-10-30T20:45:47.530050+0000","last_peered":"2023-10-30T20:45:47.530050+0000","last_clean":"2023-10-30T20:45:47.530050+0000","last_became_active":"2023-10-30T20:25:23.473157+0000","last_became_peered":"2023-10-30T20:25:23.473157+0000","last_unstale":"2023-10-30T20:45:47.530050+0000","last_undegraded":"2023-10-30T20:45:47.530050+0000","last_fullsized":"2023-10-30T20:45:47.530050+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:47:35.168516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.896471+0000","last_change":"2023-10-30T20:24:04.284202+0000","last_active":"2023-10-30T20:45:46.896471+0000","last_peered":"2023-10-30T20:45:46.896471+0000","last_clean":"2023-10-30T20:45:46.896471+0000","last_became_active":"2023-10-30T20:24:04.254051+0000","last_became_peered":"2023-10-30T20:24:04.254051+0000","last_unstale":"2023-10-30T20:45:46.896471+0000","last_undegraded":"2023-10-30T20:45:46.896471+0000","last_fullsized":"2023-10-30T20:45:46.896471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:14:06.705421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":878,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.465358+0000","last_change":"2023-10-30T20:25:36.268959+0000","last_active":"2023-10-30T20:45:47.465358+0000","last_peered":"2023-10-30T20:45:47.465358+0000","last_clean":"2023-10-30T20:45:47.465358+0000","last_became_active":"2023-10-30T20:25:36.260481+0000","last_became_peered":"2023-10-30T20:25:36.260481+0000","last_unstale":"2023-10-30T20:45:47.465358+0000","last_undegraded":"2023-10-30T20:45:47.465358+0000","last_fullsized":"2023-10-30T20:45:47.465358+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:06:50.808567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.917896+0000","last_change":"2023-10-30T20:24:04.676966+0000","last_active":"2023-10-30T20:45:46.917896+0000","last_peered":"2023-10-30T20:45:46.917896+0000","last_clean":"2023-10-30T20:45:46.917896+0000","last_became_active":"2023-10-30T20:24:04.671790+0000","last_became_peered":"2023-10-30T20:24:04.671790+0000","last_unstale":"2023-10-30T20:45:46.917896+0000","last_undegraded":"2023-10-30T20:45:46.917896+0000","last_fullsized":"2023-10-30T20:45:46.917896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:56.600896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":989,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.441313+0000","last_change":"2023-10-30T20:23:57.127569+0000","last_active":"2023-10-30T20:45:47.441313+0000","last_peered":"2023-10-30T20:45:47.441313+0000","last_clean":"2023-10-30T20:45:47.441313+0000","last_became_active":"2023-10-30T20:23:57.098872+0000","last_became_peered":"2023-10-30T20:23:57.098872+0000","last_unstale":"2023-10-30T20:45:47.441313+0000","last_undegraded":"2023-10-30T20:45:47.441313+0000","last_fullsized":"2023-10-30T20:45:47.441313+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:23:54.534733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:23:54.534733+0000","last_clean_scrub_stamp":"2023-10-30T20:23:54.534733+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:26:29.039781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"45'7","reported_seq":860,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.892575+0000","last_change":"2023-10-30T20:25:25.378575+0000","last_active":"2023-10-30T20:45:46.892575+0000","last_peered":"2023-10-30T20:45:46.892575+0000","last_clean":"2023-10-30T20:45:46.892575+0000","last_became_active":"2023-10-30T20:25:25.362636+0000","last_became_peered":"2023-10-30T20:25:25.362636+0000","last_unstale":"2023-10-30T20:45:46.892575+0000","last_undegraded":"2023-10-30T20:45:46.892575+0000","last_fullsized":"2023-10-30T20:45:46.892575+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:34.822570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":857,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.915351+0000","last_change":"2023-10-30T20:25:23.612993+0000","last_active":"2023-10-30T20:45:46.915351+0000","last_peered":"2023-10-30T20:45:46.915351+0000","last_clean":"2023-10-30T20:45:46.915351+0000","last_became_active":"2023-10-30T20:25:23.607599+0000","last_became_peered":"2023-10-30T20:25:23.607599+0000","last_unstale":"2023-10-30T20:45:46.915351+0000","last_undegraded":"2023-10-30T20:45:46.915351+0000","last_fullsized":"2023-10-30T20:45:46.915351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:25:35.997128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":867,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.899233+0000","last_change":"2023-10-30T20:25:28.335295+0000","last_active":"2023-10-30T20:45:46.899233+0000","last_peered":"2023-10-30T20:45:46.899233+0000","last_clean":"2023-10-30T20:45:46.899233+0000","last_became_active":"2023-10-30T20:25:28.326296+0000","last_became_peered":"2023-10-30T20:25:28.326296+0000","last_unstale":"2023-10-30T20:45:46.899233+0000","last_undegraded":"2023-10-30T20:45:46.899233+0000","last_fullsized":"2023-10-30T20:45:46.899233+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:41:04.620931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":855,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.483979+0000","last_change":"2023-10-30T20:25:20.677355+0000","last_active":"2023-10-30T20:45:47.483979+0000","last_peered":"2023-10-30T20:45:47.483979+0000","last_clean":"2023-10-30T20:45:47.483979+0000","last_became_active":"2023-10-30T20:25:20.671965+0000","last_became_peered":"2023-10-30T20:25:20.671965+0000","last_unstale":"2023-10-30T20:45:47.483979+0000","last_undegraded":"2023-10-30T20:45:47.483979+0000","last_fullsized":"2023-10-30T20:45:47.483979+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:55:35.886983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":867,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.936385+0000","last_change":"2023-10-30T20:25:28.987986+0000","last_active":"2023-10-30T20:45:46.936385+0000","last_peered":"2023-10-30T20:45:46.936385+0000","last_clean":"2023-10-30T20:45:46.936385+0000","last_became_active":"2023-10-30T20:25:28.979668+0000","last_became_peered":"2023-10-30T20:25:28.979668+0000","last_unstale":"2023-10-30T20:45:46.936385+0000","last_undegraded":"2023-10-30T20:45:46.936385+0000","last_fullsized":"2023-10-30T20:45:46.936385+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:22:41.686629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":867,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.902220+0000","last_change":"2023-10-30T20:25:23.069394+0000","last_active":"2023-10-30T20:45:46.902220+0000","last_peered":"2023-10-30T20:45:46.902220+0000","last_clean":"2023-10-30T20:45:46.902220+0000","last_became_active":"2023-10-30T20:25:23.058176+0000","last_became_peered":"2023-10-30T20:25:23.058176+0000","last_unstale":"2023-10-30T20:45:46.902220+0000","last_undegraded":"2023-10-30T20:45:46.902220+0000","last_fullsized":"2023-10-30T20:45:46.902220+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:33:56.491589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":870,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.940844+0000","last_change":"2023-10-30T20:25:28.823327+0000","last_active":"2023-10-30T20:45:46.940844+0000","last_peered":"2023-10-30T20:45:46.940844+0000","last_clean":"2023-10-30T20:45:46.940844+0000","last_became_active":"2023-10-30T20:25:28.798013+0000","last_became_peered":"2023-10-30T20:25:28.798013+0000","last_unstale":"2023-10-30T20:45:46.940844+0000","last_undegraded":"2023-10-30T20:45:46.940844+0000","last_fullsized":"2023-10-30T20:45:46.940844+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:35:37.042427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":870,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.931482+0000","last_change":"2023-10-30T20:25:23.723655+0000","last_active":"2023-10-30T20:45:46.931482+0000","last_peered":"2023-10-30T20:45:46.931482+0000","last_clean":"2023-10-30T20:45:46.931482+0000","last_became_active":"2023-10-30T20:25:23.709112+0000","last_became_peered":"2023-10-30T20:25:23.709112+0000","last_unstale":"2023-10-30T20:45:46.931482+0000","last_undegraded":"2023-10-30T20:45:46.931482+0000","last_fullsized":"2023-10-30T20:45:46.931482+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:06:46.124278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":847,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T20:45:44.902141+0000","last_change":"2023-10-30T20:25:28.781780+0000","last_active":"2023-10-30T20:45:44.902141+0000","last_peered":"2023-10-30T20:45:44.902141+0000","last_clean":"2023-10-30T20:45:44.902141+0000","last_became_active":"2023-10-30T20:25:28.762953+0000","last_became_peered":"2023-10-30T20:25:28.762953+0000","last_unstale":"2023-10-30T20:45:44.902141+0000","last_undegraded":"2023-10-30T20:45:44.902141+0000","last_fullsized":"2023-10-30T20:45:44.902141+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:09.371593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":866,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.918142+0000","last_change":"2023-10-30T20:25:21.917941+0000","last_active":"2023-10-30T20:45:46.918142+0000","last_peered":"2023-10-30T20:45:46.918142+0000","last_clean":"2023-10-30T20:45:46.918142+0000","last_became_active":"2023-10-30T20:25:21.911569+0000","last_became_peered":"2023-10-30T20:25:21.911569+0000","last_unstale":"2023-10-30T20:45:46.918142+0000","last_undegraded":"2023-10-30T20:45:46.918142+0000","last_fullsized":"2023-10-30T20:45:46.918142+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:32:26.723949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":873,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.897611+0000","last_change":"2023-10-30T20:29:14.877146+0000","last_active":"2023-10-30T20:45:46.897611+0000","last_peered":"2023-10-30T20:45:46.897611+0000","last_clean":"2023-10-30T20:45:46.897611+0000","last_became_active":"2023-10-30T20:29:14.860294+0000","last_became_peered":"2023-10-30T20:29:14.860294+0000","last_unstale":"2023-10-30T20:45:46.897611+0000","last_undegraded":"2023-10-30T20:45:46.897611+0000","last_fullsized":"2023-10-30T20:45:46.897611+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:31:25.822566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.460543+0000","last_change":"2023-10-30T20:25:29.021159+0000","last_active":"2023-10-30T20:45:47.460543+0000","last_peered":"2023-10-30T20:45:47.460543+0000","last_clean":"2023-10-30T20:45:47.460543+0000","last_became_active":"2023-10-30T20:25:29.004167+0000","last_became_peered":"2023-10-30T20:25:29.004167+0000","last_unstale":"2023-10-30T20:45:47.460543+0000","last_undegraded":"2023-10-30T20:45:47.460543+0000","last_fullsized":"2023-10-30T20:45:47.460543+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T08:00:40.390424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477001,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59904,"kb_used_data":6516,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311936,"statfs":{"total":96636764160,"available":96575422464,"internally_reserved":0,"allocated":6672384,"data_stored":3197595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":25.940000000000001,"5min":18.199999999999999,"15min":16.863},"min":{"1min":5.1040000000000001,"5min":5.0529999999999999,"15min":3.9740000000000002},"max":{"1min":156.071,"5min":222.095,"15min":244.999},"last":8.1519999999999992},{"interface":"front","average":{"1min":32.825000000000003,"5min":20.239000000000001,"15min":18.428000000000001},"min":{"1min":5.1020000000000003,"5min":4.9279999999999999,"15min":4.2539999999999996},"max":{"1min":219.797,"5min":219.797,"15min":219.797},"last":6.5129999999999999}]},{"osd":1,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":25.375,"5min":20.100999999999999,"15min":20.373000000000001},"min":{"1min":9.6869999999999994,"5min":5.6500000000000004,"15min":4.5940000000000003},"max":{"1min":143.87200000000001,"5min":231.85499999999999,"15min":256.815},"last":10.204000000000001},{"interface":"front","average":{"1min":22.506,"5min":19.602,"15min":22.550999999999998},"min":{"1min":7.298,"5min":5.7320000000000002,"15min":5.726},"max":{"1min":122.70399999999999,"5min":237.88300000000001,"15min":424.57999999999998},"last":14.696999999999999}]},{"osd":2,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":34.981999999999999,"5min":24.867999999999999,"15min":24.663},"min":{"1min":9.7789999999999999,"5min":6.1539999999999999,"15min":6.1539999999999999},"max":{"1min":267.81099999999998,"5min":267.81099999999998,"15min":835.96799999999996},"last":10.823},{"interface":"front","average":{"1min":37.021999999999998,"5min":26.030999999999999,"15min":23.010000000000002},"min":{"1min":10.771000000000001,"5min":10.173,"15min":8.2479999999999993},"max":{"1min":244.50399999999999,"5min":247.178,"15min":261.50900000000001},"last":13.718999999999999}]},{"osd":4,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":29.966000000000001,"5min":27.501000000000001,"15min":23.704999999999998},"min":{"1min":6.8390000000000004,"5min":6.625,"15min":5.5190000000000001},"max":{"1min":196.904,"5min":247.904,"15min":247.904},"last":9.1969999999999992},{"interface":"front","average":{"1min":29.170999999999999,"5min":25.465,"15min":24.071000000000002},"min":{"1min":7.9900000000000002,"5min":6.8620000000000001,"15min":6.3840000000000003},"max":{"1min":210.88200000000001,"5min":235.64099999999999,"15min":235.64099999999999},"last":15.352}]},{"osd":5,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":35.137999999999998,"5min":26.385999999999999,"15min":25.082999999999998},"min":{"1min":8.4100000000000001,"5min":6.726,"15min":6.726},"max":{"1min":198.76300000000001,"5min":233.28999999999999,"15min":233.28999999999999},"last":17.306999999999999},{"interface":"front","average":{"1min":46.277000000000001,"5min":31.02,"15min":24.657},"min":{"1min":11.351000000000001,"5min":10.645,"15min":7.883},"max":{"1min":227.28299999999999,"5min":244.23500000000001,"15min":244.23500000000001},"last":18.056999999999999}]},{"osd":6,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":41.406999999999996,"5min":30.888000000000002,"15min":27.672999999999998},"min":{"1min":11.547000000000001,"5min":9.1989999999999998,"15min":9.1989999999999998},"max":{"1min":269.15600000000001,"5min":269.15600000000001,"15min":269.15600000000001},"last":20.324999999999999},{"interface":"front","average":{"1min":32.853999999999999,"5min":24.702999999999999,"15min":22.295999999999999},"min":{"1min":6.093,"5min":6.093,"15min":6.093},"max":{"1min":262.53500000000003,"5min":262.53500000000003,"15min":262.53500000000003},"last":13.135}]},{"osd":7,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":29.765999999999998,"5min":27.036000000000001,"15min":24.699000000000002},"min":{"1min":9.7669999999999995,"5min":8.5570000000000004,"15min":7.3799999999999999},"max":{"1min":222.32400000000001,"5min":264.48000000000002,"15min":264.48000000000002},"last":16.132000000000001},{"interface":"front","average":{"1min":31.981999999999999,"5min":27.591000000000001,"15min":26.734999999999999},"min":{"1min":9.2919999999999998,"5min":8.9700000000000006,"15min":8.9700000000000006},"max":{"1min":233.99700000000001,"5min":245.042,"15min":245.042},"last":18.952000000000002}]}]},{"osd":1,"up_from":16,"seq":68719477001,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64036,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94307804,"statfs":{"total":96636764160,"available":96571191296,"internally_reserved":0,"allocated":6635520,"data_stored":3528420,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.762,"5min":15.228999999999999,"15min":16.553999999999998},"min":{"1min":4.1130000000000004,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":98.356999999999999,"5min":244.65000000000001,"15min":244.65000000000001},"last":4.8230000000000004},{"interface":"front","average":{"1min":11.536,"5min":12.617000000000001,"15min":18.32},"min":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":3.847},"max":{"1min":39.68,"5min":65.853999999999999,"15min":407.38900000000001},"last":9.5150000000000006}]},{"osd":2,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":18.402000000000001,"5min":16.823,"15min":19.931000000000001},"min":{"1min":7.5389999999999997,"5min":7.367,"15min":4.1500000000000004},"max":{"1min":80.688000000000002,"5min":113.649,"15min":297.26499999999999},"last":13.631},{"interface":"front","average":{"1min":23.346,"5min":19.442,"15min":20.483000000000001},"min":{"1min":9.8040000000000003,"5min":8.5690000000000008,"15min":6.5469999999999997},"max":{"1min":148.03999999999999,"5min":148.03999999999999,"15min":167.12100000000001},"last":14.256}]},{"osd":3,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":17.946000000000002,"5min":19.791,"15min":21.096},"min":{"1min":8.8569999999999993,"5min":7.8029999999999999,"15min":6.5990000000000002},"max":{"1min":84.090999999999994,"5min":150.166,"15min":213.971},"last":15.526},{"interface":"front","average":{"1min":19.449000000000002,"5min":22.609000000000002,"15min":24.210999999999999},"min":{"1min":12.093999999999999,"5min":9.8409999999999993,"15min":8.1449999999999996},"max":{"1min":74.275999999999996,"5min":146.15899999999999,"15min":223.49199999999999},"last":16.167000000000002}]},{"osd":4,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":13.090999999999999,"5min":20.369,"15min":21.076000000000001},"min":{"1min":8.1349999999999998,"5min":6.8659999999999997,"15min":5.9809999999999999},"max":{"1min":30.533000000000001,"5min":200.98400000000001,"15min":215.435},"last":10.723000000000001},{"interface":"front","average":{"1min":17.289000000000001,"5min":23.798999999999999,"15min":25.364000000000001},"min":{"1min":12.874000000000001,"5min":11.516999999999999,"15min":8.9939999999999998},"max":{"1min":26.600000000000001,"5min":192.69200000000001,"15min":197.34},"last":17.629999999999999}]},{"osd":5,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":13.349,"5min":19.747,"15min":19.356000000000002},"min":{"1min":7.4020000000000001,"5min":6.4580000000000002,"15min":5.1390000000000002},"max":{"1min":62.798000000000002,"5min":209.35499999999999,"15min":241.798},"last":7.5170000000000003},{"interface":"front","average":{"1min":17.783000000000001,"5min":20.292000000000002,"15min":21.498000000000001},"min":{"1min":9.5739999999999998,"5min":6.6230000000000002,"15min":6.6230000000000002},"max":{"1min":68.882000000000005,"5min":118.65600000000001,"15min":248.124},"last":11.372}]},{"osd":6,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.948,"5min":20.902000000000001,"15min":23.236999999999998},"min":{"1min":10.205,"5min":7.7220000000000004,"15min":6.9880000000000004},"max":{"1min":37.037999999999997,"5min":204.71899999999999,"15min":406.72399999999999},"last":12.73},{"interface":"front","average":{"1min":28.690999999999999,"5min":25.748000000000001,"15min":25.416},"min":{"1min":11.704000000000001,"5min":10.397,"15min":8.4749999999999996},"max":{"1min":179.98099999999999,"5min":179.98099999999999,"15min":179.98099999999999},"last":18.324000000000002}]},{"osd":7,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":14.643000000000001,"5min":18.478999999999999,"15min":23.765000000000001},"min":{"1min":6.0410000000000004,"5min":6.0410000000000004,"15min":6.0110000000000001},"max":{"1min":58.841999999999999,"5min":188.792,"15min":249.88300000000001},"last":8.7400000000000002},{"interface":"front","average":{"1min":19.361000000000001,"5min":21.233000000000001,"15min":25.628},"min":{"1min":12.742000000000001,"5min":8.8559999999999999,"15min":7.1360000000000001},"max":{"1min":59.718000000000004,"5min":134.89400000000001,"15min":275.36700000000002},"last":14.907}]}]},{"osd":0,"up_from":16,"seq":68719477002,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59832,"kb_used_data":6448,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312008,"statfs":{"total":96636764160,"available":96575496192,"internally_reserved":0,"allocated":6602752,"data_stored":3130585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":56,"apply_latency_ms":56,"commit_latency_ns":56000000,"apply_latency_ns":56000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":14.162000000000001,"5min":13.208,"15min":18.428000000000001},"min":{"1min":5.508,"5min":4.1740000000000004,"15min":4.0819999999999999},"max":{"1min":64.745999999999995,"5min":145.857,"15min":341.85300000000001},"last":6.968},{"interface":"front","average":{"1min":15.507,"5min":16.523,"15min":18.965},"min":{"1min":5.9400000000000004,"5min":5.798,"15min":5.1669999999999998},"max":{"1min":71.578000000000003,"5min":196.148,"15min":243.62100000000001},"last":21.934999999999999}]},{"osd":2,"last update":"Mon Oct 30 20:44:59 2023","interfaces":[{"interface":"back","average":{"1min":18.457000000000001,"5min":19.640999999999998,"15min":31.206},"min":{"1min":6.5940000000000003,"5min":4.6449999999999996,"15min":4.6449999999999996},"max":{"1min":68.856999999999999,"5min":199.71100000000001,"15min":580.43100000000004},"last":6.0309999999999997},{"interface":"front","average":{"1min":15.906000000000001,"5min":18.309000000000001,"15min":31.786999999999999},"min":{"1min":6.6920000000000002,"5min":6.6920000000000002,"15min":6.1420000000000003},"max":{"1min":55.250999999999998,"5min":197.125,"15min":1090.171},"last":15.888}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.609,"5min":16.076000000000001,"15min":16.977},"min":{"1min":6.2199999999999998,"5min":5.298,"15min":5.298},"max":{"1min":76.861000000000004,"5min":148.31299999999999,"15min":167.607},"last":8.2200000000000006},{"interface":"front","average":{"1min":30.317,"5min":25.641999999999999,"15min":24.917999999999999},"min":{"1min":7.1719999999999997,"5min":7.1719999999999997,"15min":6.4370000000000003},"max":{"1min":290.19600000000003,"5min":290.19600000000003,"15min":318.245},"last":23.213999999999999}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":24.504000000000001,"5min":20.890000000000001,"15min":20.295000000000002},"min":{"1min":7.234,"5min":7.1059999999999999,"15min":5.8170000000000002},"max":{"1min":116.98399999999999,"5min":174.67699999999999,"15min":178.642},"last":11.069000000000001},{"interface":"front","average":{"1min":30.776,"5min":28.367999999999999,"15min":27.843},"min":{"1min":8.3770000000000007,"5min":8.1010000000000009,"15min":6.2960000000000003},"max":{"1min":153.124,"5min":219.14699999999999,"15min":242.06800000000001},"last":33.905999999999999}]},{"osd":5,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.25,"5min":23.315999999999999,"15min":22.43},"min":{"1min":9.3379999999999992,"5min":8.3480000000000008,"15min":6.8099999999999996},"max":{"1min":68.546999999999997,"5min":170.024,"15min":170.024},"last":26.283999999999999},{"interface":"front","average":{"1min":25.843,"5min":23.294,"15min":22.381},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.8280000000000003},"max":{"1min":138.041,"5min":177.41800000000001,"15min":177.41800000000001},"last":25.364999999999998}]},{"osd":6,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":27.763999999999999,"5min":29.074999999999999,"15min":30.167000000000002},"min":{"1min":10.817,"5min":8.7599999999999998,"15min":8.2590000000000003},"max":{"1min":151.73500000000001,"5min":189.21899999999999,"15min":330.37799999999999},"last":34.863999999999997},{"interface":"front","average":{"1min":19.260000000000002,"5min":20.901,"15min":24.004000000000001},"min":{"1min":6.8410000000000002,"5min":6.8410000000000002,"15min":6.8410000000000002},"max":{"1min":73.170000000000002,"5min":169.447,"15min":201.50299999999999},"last":28.603999999999999}]},{"osd":7,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":20.75,"5min":23.949999999999999,"15min":27.219999999999999},"min":{"1min":11.298,"5min":10.81,"15min":9.6820000000000004},"max":{"1min":70.683000000000007,"5min":175.667,"15min":268.18900000000002},"last":30.404},{"interface":"front","average":{"1min":21.437999999999999,"5min":21.204999999999998,"15min":25.626000000000001},"min":{"1min":8.8689999999999998,"5min":8.8689999999999998,"15min":7.3559999999999999},"max":{"1min":77.927000000000007,"5min":173.441,"15min":228.495},"last":27.532}]}]},{"osd":6,"up_from":15,"seq":64424509705,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59840,"kb_used_data":6444,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312000,"statfs":{"total":96636764160,"available":96575488000,"internally_reserved":0,"allocated":6598656,"data_stored":3155698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":54,"apply_latency_ms":54,"commit_latency_ns":54000000,"apply_latency_ns":54000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:44:48 2023","interfaces":[{"interface":"back","average":{"1min":13.409000000000001,"5min":11.871,"15min":16.204999999999998},"min":{"1min":4.3979999999999997,"5min":4.0179999999999998,"15min":3.7080000000000002},"max":{"1min":73.316000000000003,"5min":73.316000000000003,"15min":158.14599999999999},"last":12.824999999999999},{"interface":"front","average":{"1min":10.07,"5min":12.122,"15min":19.719000000000001},"min":{"1min":5.407,"5min":5.2439999999999998,"15min":4.5499999999999998},"max":{"1min":54.936999999999998,"5min":75.060000000000002,"15min":488.43799999999999},"last":17.391999999999999}]},{"osd":1,"last update":"Mon Oct 30 20:44:48 2023","interfaces":[{"interface":"back","average":{"1min":13.942,"5min":15.932,"15min":20.992999999999999},"min":{"1min":5.4379999999999997,"5min":5.4379999999999997,"15min":4.9199999999999999},"max":{"1min":38.279000000000003,"5min":117.193,"15min":218.78100000000001},"last":6.5860000000000003},{"interface":"front","average":{"1min":15.507,"5min":14.162000000000001,"15min":16.847000000000001},"min":{"1min":5.3230000000000004,"5min":5.1379999999999999,"15min":5.0149999999999997},"max":{"1min":103.065,"5min":128.50200000000001,"15min":155.27199999999999},"last":10.08}]},{"osd":2,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":19.477,"5min":16.728000000000002,"15min":21.635000000000002},"min":{"1min":7.9080000000000004,"5min":6.1870000000000003,"15min":6.1870000000000003},"max":{"1min":67.552999999999997,"5min":122.307,"15min":222.303},"last":10.866},{"interface":"front","average":{"1min":16.923999999999999,"5min":18.561,"15min":26.841000000000001},"min":{"1min":7.9580000000000002,"5min":7.9580000000000002,"15min":6.1820000000000004},"max":{"1min":46.755000000000003,"5min":98.781000000000006,"15min":438.19200000000001},"last":11.613}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.385000000000002,"5min":15.81,"15min":17.850000000000001},"min":{"1min":6.4859999999999998,"5min":6.2050000000000001,"15min":6.0179999999999998},"max":{"1min":136.16800000000001,"5min":136.16800000000001,"15min":217.99000000000001},"last":7.3499999999999996},{"interface":"front","average":{"1min":30.402000000000001,"5min":20.652999999999999,"15min":21.940000000000001},"min":{"1min":6.3890000000000002,"5min":6.3890000000000002,"15min":6.0010000000000003},"max":{"1min":270.25599999999997,"5min":270.25599999999997,"15min":270.25599999999997},"last":15.512}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.241,"5min":21.978000000000002,"15min":23.396000000000001},"min":{"1min":11.669,"5min":8.1539999999999999,"15min":7.5800000000000001},"max":{"1min":69.150000000000006,"5min":168.38800000000001,"15min":190.29599999999999},"last":34.412999999999997},{"interface":"front","average":{"1min":14.085000000000001,"5min":20.285,"15min":22.73},"min":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"max":{"1min":48.207000000000001,"5min":163.41800000000001,"15min":189.59399999999999},"last":9.3260000000000005}]},{"osd":5,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":21.626999999999999,"5min":20.792999999999999,"15min":22.007999999999999},"min":{"1min":9.1430000000000007,"5min":8.9939999999999998,"15min":8.9930000000000003},"max":{"1min":68.114000000000004,"5min":133.73400000000001,"15min":138.38800000000001},"last":24.062000000000001},{"interface":"front","average":{"1min":25.091000000000001,"5min":24.527000000000001,"15min":25.061},"min":{"1min":12.673999999999999,"5min":11.9,"15min":10.571},"max":{"1min":68.138999999999996,"5min":150.35499999999999,"15min":152.40199999999999},"last":19.061}]},{"osd":7,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":17.923999999999999,"5min":18.893999999999998,"15min":23.824999999999999},"min":{"1min":8.8140000000000001,"5min":7.1699999999999999,"15min":6.226},"max":{"1min":64.828999999999994,"5min":98.503,"15min":253.16900000000001},"last":16.713000000000001},{"interface":"front","average":{"1min":19.021000000000001,"5min":21.308,"15min":20.143000000000001},"min":{"1min":7.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"max":{"1min":75.846000000000004,"5min":174.071,"15min":221.28399999999999},"last":12.241}]}]},{"osd":5,"up_from":15,"seq":64424509703,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77920,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293920,"statfs":{"total":96636764160,"available":96556974080,"internally_reserved":0,"allocated":6635520,"data_stored":3528420,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":17.625,"5min":18.785,"15min":16.260999999999999},"min":{"1min":5.2329999999999997,"5min":3.8359999999999999,"15min":3.8359999999999999},"max":{"1min":85.513999999999996,"5min":220.148,"15min":220.148},"last":10.542999999999999},{"interface":"front","average":{"1min":27.861000000000001,"5min":23.675000000000001,"15min":20.949000000000002},"min":{"1min":8.6039999999999992,"5min":5.6059999999999999,"15min":5.6059999999999999},"max":{"1min":145.43000000000001,"5min":177.91,"15min":177.91},"last":14.894}]},{"osd":1,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":20.715,"5min":16.585000000000001,"15min":20.353999999999999},"min":{"1min":6.3040000000000003,"5min":4.3010000000000002,"15min":4.3010000000000002},"max":{"1min":71.460999999999999,"5min":74.081000000000003,"15min":293.75400000000002},"last":9.6920000000000002},{"interface":"front","average":{"1min":20.943999999999999,"5min":16.175000000000001,"15min":18.327999999999999},"min":{"1min":5.46,"5min":4.8949999999999996,"15min":4.8949999999999996},"max":{"1min":80.397000000000006,"5min":80.397000000000006,"15min":210.797},"last":12.271000000000001}]},{"osd":2,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":31.280000000000001,"5min":23.431999999999999,"15min":25.231000000000002},"min":{"1min":6.2480000000000002,"5min":5.0209999999999999,"15min":5.0209999999999999},"max":{"1min":105.449,"5min":143.273,"15min":519.68799999999999},"last":13.766},{"interface":"front","average":{"1min":23.294,"5min":20.007000000000001,"15min":22.029},"min":{"1min":5.6479999999999997,"5min":5.6479999999999997,"15min":5.226},"max":{"1min":82.042000000000002,"5min":154.083,"15min":218.59899999999999},"last":6.0469999999999997}]},{"osd":3,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":41.817,"5min":31.161999999999999,"15min":26.263000000000002},"min":{"1min":9.9900000000000002,"5min":9.9900000000000002,"15min":7.6150000000000002},"max":{"1min":269.52999999999997,"5min":269.52999999999997,"15min":269.52999999999997},"last":16.454999999999998},{"interface":"front","average":{"1min":24.488,"5min":22.629000000000001,"15min":22.704000000000001},"min":{"1min":7.4480000000000004,"5min":6.218,"15min":5.9119999999999999},"max":{"1min":178.78700000000001,"5min":178.78700000000001,"15min":249.797},"last":6.7670000000000003}]},{"osd":4,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":21.725999999999999,"5min":26.437999999999999,"15min":25.25},"min":{"1min":8.1080000000000005,"5min":7.1580000000000004,"15min":6.6100000000000003},"max":{"1min":78.555999999999997,"5min":215.97999999999999,"15min":291.92599999999999},"last":8.1080000000000005},{"interface":"front","average":{"1min":26.097000000000001,"5min":33.600000000000001,"15min":28.585000000000001},"min":{"1min":11.895,"5min":8.0050000000000008,"15min":6.9550000000000001},"max":{"1min":70.290999999999997,"5min":324.31,"15min":324.31},"last":13.007999999999999}]},{"osd":6,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":32.883000000000003,"5min":28.116,"15min":26.379999999999999},"min":{"1min":10.593,"5min":8.6489999999999991,"15min":7.2199999999999998},"max":{"1min":162.964,"5min":211.255,"15min":251.37700000000001},"last":8.7439999999999998},{"interface":"front","average":{"1min":38.536000000000001,"5min":33.917999999999999,"15min":28.524000000000001},"min":{"1min":9.3640000000000008,"5min":9.2349999999999994,"15min":7.1619999999999999},"max":{"1min":160.42599999999999,"5min":333.29700000000003,"15min":333.29700000000003},"last":17.245000000000001}]},{"osd":7,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":36.252000000000002,"5min":33.210000000000001,"15min":27.071999999999999},"min":{"1min":11.131,"5min":10.68,"15min":6.1769999999999996},"max":{"1min":116.877,"5min":258.81,"15min":264.52999999999997},"last":15.683999999999999},{"interface":"front","average":{"1min":26.748000000000001,"5min":22.239000000000001,"15min":25.803999999999998},"min":{"1min":7.4580000000000002,"5min":5.8410000000000002,"15min":5.8410000000000002},"max":{"1min":90.570999999999998,"5min":154.86799999999999,"15min":255.625},"last":11.167999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542408,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59852,"kb_used_data":6456,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311988,"statfs":{"total":96636764160,"available":96575475712,"internally_reserved":0,"allocated":6610944,"data_stored":3136155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":9.2010000000000005,"5min":13.369999999999999,"15min":14.285},"min":{"1min":4.9580000000000002,"5min":4.0499999999999998,"15min":3.5920000000000001},"max":{"1min":18.266999999999999,"5min":106.79300000000001,"15min":187.85499999999999},"last":18.699999999999999},{"interface":"front","average":{"1min":8.7729999999999997,"5min":12.766999999999999,"15min":13.725},"min":{"1min":5.0279999999999996,"5min":4.1020000000000003,"15min":4.1020000000000003},"max":{"1min":15.933,"5min":117.645,"15min":124.342},"last":7.9589999999999996}]},{"osd":1,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":13.706,"5min":18.475999999999999,"15min":19.167999999999999},"min":{"1min":8.1969999999999992,"5min":5.8979999999999997,"15min":5.7409999999999997},"max":{"1min":33.970999999999997,"5min":103.907,"15min":226.63399999999999},"last":19.806000000000001},{"interface":"front","average":{"1min":11.929,"5min":17.565999999999999,"15min":19.605},"min":{"1min":6.5439999999999996,"5min":6.351,"15min":5.859},"max":{"1min":42.719000000000001,"5min":135.41200000000001,"15min":187.24100000000001},"last":11.147}]},{"osd":2,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":17.675000000000001,"5min":18.248999999999999,"15min":20.965},"min":{"1min":7.2869999999999999,"5min":6.2549999999999999,"15min":6.2549999999999999},"max":{"1min":43.064999999999998,"5min":79.168000000000006,"15min":200.285},"last":13.004},{"interface":"front","average":{"1min":26.209,"5min":21.07,"15min":21.149999999999999},"min":{"1min":6.0679999999999996,"5min":5.5090000000000003,"15min":5.5090000000000003},"max":{"1min":211.315,"5min":211.315,"15min":241.15700000000001},"last":5.4720000000000004}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":25.942,"5min":21.164000000000001,"15min":22.437999999999999},"min":{"1min":10.516,"5min":8.1509999999999998,"15min":6.6849999999999996},"max":{"1min":205.821,"5min":205.821,"15min":262.43900000000002},"last":14.073},{"interface":"front","average":{"1min":26.780000000000001,"5min":22.042999999999999,"15min":23.443999999999999},"min":{"1min":7.5650000000000004,"5min":6.7460000000000004,"15min":5.9950000000000001},"max":{"1min":225.29900000000001,"5min":225.29900000000001,"15min":330.15300000000002},"last":10.228}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.153,"5min":21.227,"15min":22.390000000000001},"min":{"1min":9.9459999999999997,"5min":7.218,"15min":7.218},"max":{"1min":22.241,"5min":136.494,"15min":213.072},"last":15.015000000000001},{"interface":"front","average":{"1min":17.73,"5min":22.32,"15min":24.960999999999999},"min":{"1min":8.9540000000000006,"5min":8.9540000000000006,"15min":8.5530000000000008},"max":{"1min":28.087,"5min":104.538,"15min":239.76900000000001},"last":17.381}]},{"osd":5,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.374000000000001,"5min":18.962,"15min":22.965},"min":{"1min":10.206,"5min":8.2889999999999997,"15min":7.258},"max":{"1min":27.122,"5min":110.45099999999999,"15min":275.08100000000002},"last":12.374000000000001},{"interface":"front","average":{"1min":13.379,"5min":15.702999999999999,"15min":18.018000000000001},"min":{"1min":5.3410000000000002,"5min":5.3410000000000002,"15min":5.3410000000000002},"max":{"1min":40.390000000000001,"5min":94.302999999999997,"15min":129.935},"last":7.3300000000000001}]},{"osd":6,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":18.283000000000001,"5min":21.891999999999999,"15min":25.209},"min":{"1min":11.725,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":27.867999999999999,"5min":126.001,"15min":263.41399999999999},"last":16.436},{"interface":"front","average":{"1min":17.962,"5min":19.896000000000001,"15min":23.355},"min":{"1min":11.305999999999999,"5min":8.0229999999999997,"15min":8.0229999999999997},"max":{"1min":56.701000000000001,"5min":111.691,"15min":257.75900000000001},"last":20.581}]}]},{"osd":2,"up_from":14,"seq":60129542408,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82048,"kb_used_data":6448,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289792,"statfs":{"total":96636764160,"available":96552747008,"internally_reserved":0,"allocated":6602752,"data_stored":3130585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":65,"apply_latency_ms":65,"commit_latency_ns":65000000,"apply_latency_ns":65000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":19.824999999999999,"5min":15.673,"15min":19.306999999999999},"min":{"1min":4.6959999999999997,"5min":4.3170000000000002,"15min":4.3170000000000002},"max":{"1min":211.80500000000001,"5min":211.80500000000001,"15min":389.31400000000002},"last":269.779},{"interface":"front","average":{"1min":18.757999999999999,"5min":20.440999999999999,"15min":19.539000000000001},"min":{"1min":5.9269999999999996,"5min":5.3040000000000003,"15min":4.4589999999999996},"max":{"1min":213.93600000000001,"5min":295.87,"15min":295.87},"last":124.283}]},{"osd":1,"last update":"Mon Oct 30 20:45:48 2023","interfaces":[{"interface":"back","average":{"1min":14.757,"5min":16.274999999999999,"15min":22.295999999999999},"min":{"1min":6.4720000000000004,"5min":5.5060000000000002,"15min":5.2649999999999997},"max":{"1min":36.689,"5min":74.575000000000003,"15min":245.84999999999999},"last":9.8390000000000004},{"interface":"front","average":{"1min":17.843,"5min":15.342000000000001,"15min":20.946999999999999},"min":{"1min":7.9459999999999997,"5min":5.8399999999999999,"15min":5.3330000000000002},"max":{"1min":66.947999999999993,"5min":72.662999999999997,"15min":317.20699999999999},"last":13.996}]},{"osd":3,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.094000000000001,"5min":17.843,"15min":22.158000000000001},"min":{"1min":6.6040000000000001,"5min":6.1459999999999999,"15min":5.2619999999999996},"max":{"1min":58.442999999999998,"5min":253.02799999999999,"15min":311.03100000000001},"last":8.6120000000000001},{"interface":"front","average":{"1min":20.954000000000001,"5min":21.210999999999999,"15min":28.140999999999998},"min":{"1min":10.403,"5min":7.4969999999999999,"15min":6.2039999999999997},"max":{"1min":86.935000000000002,"5min":145.066,"15min":703},"last":10.464}]},{"osd":4,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":14.718,"5min":18.241,"15min":26.344999999999999},"min":{"1min":7.4029999999999996,"5min":6.4809999999999999,"15min":6.3460000000000001},"max":{"1min":43.838999999999999,"5min":123.06399999999999,"15min":709.85900000000004},"last":12.739000000000001},{"interface":"front","average":{"1min":16.712,"5min":19.233000000000001,"15min":22.349},"min":{"1min":6.2359999999999998,"5min":6.2359999999999998,"15min":6.2359999999999998},"max":{"1min":65.897000000000006,"5min":118.82599999999999,"15min":182.947},"last":28.228000000000002}]},{"osd":5,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":18.765000000000001,"5min":18.805,"15min":29.754000000000001},"min":{"1min":8.3539999999999992,"5min":7.2279999999999998,"15min":6.8460000000000001},"max":{"1min":63.963000000000001,"5min":116.352,"15min":874.22799999999995},"last":31.942},{"interface":"front","average":{"1min":17.289999999999999,"5min":18.369,"15min":24.251999999999999},"min":{"1min":6.9500000000000002,"5min":6.9500000000000002,"15min":6.9500000000000002},"max":{"1min":41.155000000000001,"5min":73.218999999999994,"15min":222.76400000000001},"last":26.239999999999998}]},{"osd":6,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":15.76,"5min":23.898,"15min":25.614999999999998},"min":{"1min":8.5389999999999997,"5min":8.5389999999999997,"15min":7.3049999999999997},"max":{"1min":37.296999999999997,"5min":196.654,"15min":238.84800000000001},"last":28.916},{"interface":"front","average":{"1min":18.132000000000001,"5min":23.280000000000001,"15min":29.577000000000002},"min":{"1min":8.3770000000000007,"5min":8.0670000000000002,"15min":7.577},"max":{"1min":37.128999999999998,"5min":161.30799999999999,"15min":266.48599999999999},"last":31.02}]},{"osd":7,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":19.760000000000002,"5min":20.834,"15min":26.890000000000001},"min":{"1min":9.7300000000000004,"5min":9.7300000000000004,"15min":9.7300000000000004},"max":{"1min":39.747,"5min":133.429,"15min":354.904},"last":30.41},{"interface":"front","average":{"1min":19.949999999999999,"5min":21.204000000000001,"15min":29.919},"min":{"1min":9.1600000000000001,"5min":7.3440000000000003,"15min":7.3440000000000003},"max":{"1min":75.688999999999993,"5min":153.803,"15min":865.25800000000004},"last":13.436}]}]},{"osd":4,"up_from":13,"seq":55834575114,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82072,"kb_used_data":6476,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289768,"statfs":{"total":96636764160,"available":96552722432,"internally_reserved":0,"allocated":6631424,"data_stored":3163348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":59,"apply_latency_ms":59,"commit_latency_ns":59000000,"apply_latency_ns":59000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":12.099,"5min":10.295,"15min":15.339},"min":{"1min":4.5940000000000003,"5min":3.8839999999999999,"15min":3.839},"max":{"1min":73.688000000000002,"5min":118.02500000000001,"15min":224.95099999999999},"last":8.1910000000000007},{"interface":"front","average":{"1min":14.191000000000001,"5min":11.253,"15min":15.319000000000001},"min":{"1min":4.407,"5min":4.407,"15min":4.0229999999999997},"max":{"1min":70.055999999999997,"5min":70.055999999999997,"15min":227.71600000000001},"last":7.0209999999999999}]},{"osd":1,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":19.260999999999999,"5min":15.923,"15min":20.707999999999998},"min":{"1min":6.3550000000000004,"5min":4.7960000000000003,"15min":4.7960000000000003},"max":{"1min":92.763000000000005,"5min":118.747,"15min":383.57499999999999},"last":7.4740000000000002},{"interface":"front","average":{"1min":20.491,"5min":19.440000000000001,"15min":22.885999999999999},"min":{"1min":8.3000000000000007,"5min":7.8970000000000002,"15min":7.5},"max":{"1min":59.844999999999999,"5min":110.129,"15min":228.34999999999999},"last":10.853999999999999}]},{"osd":2,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":17.518000000000001,"5min":13.864000000000001,"15min":21.329000000000001},"min":{"1min":8.2390000000000008,"5min":4.6420000000000003,"15min":4.6420000000000003},"max":{"1min":54.417999999999999,"5min":82.471999999999994,"15min":347.30799999999999},"last":15.742000000000001},{"interface":"front","average":{"1min":23.317,"5min":16.451000000000001,"15min":21.233000000000001},"min":{"1min":6.1369999999999996,"5min":6.1369999999999996,"15min":6.0090000000000003},"max":{"1min":121.45999999999999,"5min":121.45999999999999,"15min":183.03100000000001},"last":11.478}]},{"osd":3,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":16.106000000000002,"5min":17.736000000000001,"15min":24.536000000000001},"min":{"1min":6.4779999999999998,"5min":6.4779999999999998,"15min":6.2930000000000001},"max":{"1min":70.343999999999994,"5min":174.733,"15min":270.19400000000002},"last":12.321999999999999},{"interface":"front","average":{"1min":18.103999999999999,"5min":17.949999999999999,"15min":24.756},"min":{"1min":6.0220000000000002,"5min":6.0220000000000002,"15min":6.0220000000000002},"max":{"1min":63.222000000000001,"5min":177.87,"15min":339.79700000000003},"last":9.3510000000000009}]},{"osd":5,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":20.681000000000001,"5min":20.640999999999998,"15min":24.623999999999999},"min":{"1min":6.5839999999999996,"5min":6.5839999999999996,"15min":6.5839999999999996},"max":{"1min":158.541,"5min":158.541,"15min":253.80199999999999},"last":16.981999999999999},{"interface":"front","average":{"1min":17.707000000000001,"5min":16.873000000000001,"15min":22.225000000000001},"min":{"1min":6.1029999999999998,"5min":6.1029999999999998,"15min":6.1029999999999998},"max":{"1min":147.54400000000001,"5min":147.54400000000001,"15min":271.572},"last":13.77}]},{"osd":6,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":26.273,"5min":18.638999999999999,"15min":24.821999999999999},"min":{"1min":7.0999999999999996,"5min":6.4210000000000003,"15min":6.4210000000000003},"max":{"1min":245.447,"5min":245.447,"15min":254.44399999999999},"last":15.035},{"interface":"front","average":{"1min":18.928999999999998,"5min":19.975000000000001,"15min":28.715},"min":{"1min":10.25,"5min":9.0419999999999998,"15min":8.1639999999999997},"max":{"1min":72.599000000000004,"5min":74.463999999999999,"15min":291.62599999999998},"last":14.419}]},{"osd":7,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":14.513,"5min":16.260000000000002,"15min":22.474},"min":{"1min":8.1890000000000001,"5min":8.1890000000000001,"15min":6.2389999999999999},"max":{"1min":43.911999999999999,"5min":76.325000000000003,"15min":262.82100000000003},"last":17.632999999999999},{"interface":"front","average":{"1min":19.515000000000001,"5min":19.870999999999999,"15min":25.632000000000001},"min":{"1min":9.8350000000000009,"5min":9.8350000000000009,"15min":9.8350000000000009},"max":{"1min":61.091000000000001,"5min":61.091000000000001,"15min":244.37799999999999},"last":18.791}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T20:45:52.385 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T20:45:52.385 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T20:45:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.483+0000 7f2e95a02640 10 monclient: get_monmap_and_config 2023-10-30T20:45:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.483+0000 7f2e95a02640 10 monclient: build_initial_monmap 2023-10-30T20:45:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.483+0000 7f2e95a02640 10 monclient: monmap: 2023-10-30T20:45:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:52.486 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:52.486 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:52.484793+0000 2023-10-30T20:45:52.486 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:52.484793+0000 2023-10-30T20:45:52.486 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:52.488 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:52.488 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:52.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.484+0000 7f2e95a02640 1 Processor -- start 2023-10-30T20:45:52.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.484+0000 7f2e95a02640 1 -- start start 2023-10-30T20:45:52.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.484+0000 7f2e95a02640 10 monclient: init 2023-10-30T20:45:52.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e95a02640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:52.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e95a02640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-10-30T20:45:52.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e95a02640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e900aea90 addr v1:172.21.15.183:6789/0 2023-10-30T20:45:52.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e95a02640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2e900b12c0 0x7f2e900ad520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:52.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e95a02640 10 monclient(hunting): picked mon.noname-c con 0x7f2e900b12c0 addr v2:172.21.15.183:3300/0 2023-10-30T20:45:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e95a02640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e900ada60 addr v1:172.21.15.69:6789/0 2023-10-30T20:45:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e95a02640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e900b0e20 con 0x7f2e900ada60 2023-10-30T20:45:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e95a02640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e900afe80 con 0x7f2e900aea90 2023-10-30T20:45:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e95a02640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e95a02640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e900679a0 con 0x7f2e900b12c0 2023-10-30T20:45:52.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e95a02640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:52.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e95a02640 10 monclient(hunting): authenticate will time out at 2875.773958s 2023-10-30T20:45:52.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e8f7fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2e900ada60 0x7f2e9014c210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:38900/0 (socket says 172.21.15.69:38900) 2023-10-30T20:45:52.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e8f7fe640 1 -- 172.21.15.69:0/2535802866 learned_addr learned my addr 172.21.15.69:0/2535802866 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:45:52.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.485+0000 7f2e8e7fc640 1 --2- 172.21.15.69:0/2535802866 >> v2:172.21.15.183:3300/0 conn(0x7f2e900b12c0 0x7f2e900ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:52.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.486+0000 7f2e8e7fc640 10 monclient(hunting): get_auth_request con 0x7f2e900b12c0 auth_method 0 2023-10-30T20:45:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.486+0000 7f2e8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.486+0000 7f2e8e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.486+0000 7f2e8e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.487+0000 7f2e8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.487+0000 7f2e8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:52.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.487+0000 7f2e8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:52.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.487+0000 7f2e8dffb640 1 -- 172.21.15.69:0/2535802866 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103400592 0 0) 0x7f2e88001110 con 0x7f2e900ada60 2023-10-30T20:45:52.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.488+0000 7f2e8dffb640 10 monclient(hunting): _negotiate 2023-10-30T20:45:52.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.488+0000 7f2e8dffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:52.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.488+0000 7f2e8dffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.488+0000 7f2e8dffb640 10 monclient(hunting): my global_id is 16224 2023-10-30T20:45:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.488+0000 7f2e8dffb640 1 -- 172.21.15.69:0/2535802866 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e78003520 con 0x7f2e900ada60 2023-10-30T20:45:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.488+0000 7f2e8e7fc640 10 monclient(hunting): handle_auth_done global_id 21202 payload 274 2023-10-30T20:45:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.488+0000 7f2e8e7fc640 1 -- 172.21.15.69:0/2535802866 >> v1:172.21.15.183:6789/0 conn(0x7f2e900aea90 legacy=0x7f2e90141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.488+0000 7f2e8e7fc640 1 -- 172.21.15.69:0/2535802866 >> v1:172.21.15.69:6789/0 conn(0x7f2e900ada60 legacy=0x7f2e9014c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.488+0000 7f2e8e7fc640 10 monclient: _finish_hunting 0 2023-10-30T20:45:52.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.488+0000 7f2e8e7fc640 1 monclient: found mon.noname-c 2023-10-30T20:45:52.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.488+0000 7f2e8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:52.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.488+0000 7f2e8e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:45:52.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.488+0000 7f2e8e7fc640 1 -- 172.21.15.69:0/2535802866 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e900b0160 con 0x7f2e900b12c0 2023-10-30T20:45:52.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.492+0000 7f2e8e7fc640 1 --2- 172.21.15.69:0/2535802866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e900b12c0 0x7f2e900ad520 secure :-1 s=READY pgs=5043 cs=0 l=1 rev1=1 crypto rx=0x7f2e840097a0 tx=0x7f2e8402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4f4ea26a2a13007 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:52.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e8dffb640 1 -- 172.21.15.69:0/2535802866 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e84031830 con 0x7f2e900b12c0 2023-10-30T20:45:52.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:52.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e8dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:45:52.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e8dffb640 10 monclient: dump: 2023-10-30T20:45:52.500 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:52.500 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:52.500 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:52.500 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:52.500 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:52.501 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:52.501 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:52.501 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:52.501 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:52.501 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:52.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e8dffb640 10 monclient: _finish_auth 0 2023-10-30T20:45:52.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e8dffb640 10 monclient: _check_auth_tickets 2023-10-30T20:45:52.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:52.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e8dffb640 1 -- 172.21.15.69:0/2535802866 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8402d040 con 0x7f2e900b12c0 2023-10-30T20:45:52.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e95a02640 5 monclient: authenticate success, global_id 21202 2023-10-30T20:45:52.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e95a02640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:45:52.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e8dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:52.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e8dffb640 1 -- 172.21.15.69:0/2535802866 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e84031bc0 con 0x7f2e900b12c0 2023-10-30T20:45:52.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e95a02640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:52.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e95a02640 4 set_mon_vals no callback set 2023-10-30T20:45:52.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:52.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e8dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:52.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e8dffb640 10 monclient: dump: 2023-10-30T20:45:52.504 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:52.504 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:52.505 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:52.505 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:52.505 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:52.505 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:52.505 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:52.506 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:52.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e95a02640 10 monclient: shutdown 2023-10-30T20:45:52.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e95a02640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e95a02640 1 -- 172.21.15.69:0/2535802866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e900b12c0 msgr2=0x7f2e900ad520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e95a02640 1 --2- 172.21.15.69:0/2535802866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e900b12c0 0x7f2e900ad520 secure :-1 s=READY pgs=5043 cs=0 l=1 rev1=1 crypto rx=0x7f2e840097a0 tx=0x7f2e8402b8d0 comp rx=0 tx=0).stop 2023-10-30T20:45:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e95a02640 1 -- 172.21.15.69:0/2535802866 shutdown_connections 2023-10-30T20:45:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e95a02640 1 --2- 172.21.15.69:0/2535802866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e900b12c0 0x7f2e900ad520 unknown :-1 s=CLOSED pgs=5043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e95a02640 1 -- 172.21.15.69:0/2535802866 >> 172.21.15.69:0/2535802866 conn(0x7f2e900b7120 msgr2=0x7f2e900b7500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e95a02640 1 -- 172.21.15.69:0/2535802866 shutdown_connections 2023-10-30T20:45:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.498+0000 7f2e95a02640 1 -- 172.21.15.69:0/2535802866 wait complete. 2023-10-30T20:45:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.499+0000 7f2e95a02640 10 monclient: build_initial_monmap 2023-10-30T20:45:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.499+0000 7f2e95a02640 10 monclient: monmap: 2023-10-30T20:45:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:52.513 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:52.513 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:52.513 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:52.513 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:52.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.499+0000 7f2e95a02640 1 librados: starting msgr at 2023-10-30T20:45:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.499+0000 7f2e95a02640 1 librados: starting objecter 2023-10-30T20:45:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.499+0000 7f2e95a02640 1 Processor -- start 2023-10-30T20:45:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 1 -- start start 2023-10-30T20:45:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 1 librados: setting wanted keys 2023-10-30T20:45:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 1 librados: calling monclient init 2023-10-30T20:45:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 10 monclient: init 2023-10-30T20:45:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T20:45:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e900ada60 0x7f2e90144f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 10 monclient(hunting): picked mon.noname-a con 0x7f2e900ada60 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e900aea90 0x7f2e900aa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:52.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 10 monclient(hunting): picked mon.noname-c con 0x7f2e900aea90 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:52.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e900b12c0 0x7f2e900a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:52.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 10 monclient(hunting): picked mon.noname-b con 0x7f2e900b12c0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:52.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:52.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e90142990 con 0x7f2e900ada60 2023-10-30T20:45:52.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:52.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e900aab80 con 0x7f2e900b12c0 2023-10-30T20:45:52.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:52.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e90145830 con 0x7f2e900aea90 2023-10-30T20:45:52.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:52.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.500+0000 7f2e95a02640 10 monclient(hunting): authenticate will time out at 2875.789354s 2023-10-30T20:45:52.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e900ada60 0x7f2e90144f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:52.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8e7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e900aea90 0x7f2e900aa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:52.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e900ada60 0x7f2e90144f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50602/0 (socket says 172.21.15.69:50602) 2023-10-30T20:45:52.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8effd640 1 -- 172.21.15.69:0/2165832101 learned_addr learned my addr 172.21.15.69:0/2165832101 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:52.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8f7fe640 1 --2- 172.21.15.69:0/2165832101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e900b12c0 0x7f2e900a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:52.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8effd640 10 monclient(hunting): get_auth_request con 0x7f2e900ada60 auth_method 0 2023-10-30T20:45:52.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:52.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:52.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:52.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8e7fc640 10 monclient(hunting): get_auth_request con 0x7f2e900aea90 auth_method 0 2023-10-30T20:45:52.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:52.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:52.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:52.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8f7fe640 10 monclient(hunting): get_auth_request con 0x7f2e900b12c0 auth_method 0 2023-10-30T20:45:52.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:52.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:52.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:52.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:52.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:52.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:52.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:52.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:52.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.501+0000 7f2e8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:52.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.502+0000 7f2e8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:52.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.502+0000 7f2e8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:52.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.502+0000 7f2e8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:52.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.504+0000 7f2e8effd640 10 monclient(hunting): handle_auth_done global_id 16227 payload 824 2023-10-30T20:45:52.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.504+0000 7f2e8effd640 1 -- 172.21.15.69:0/2165832101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e900aea90 msgr2=0x7f2e900aa420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:52.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.504+0000 7f2e8effd640 1 --2- 172.21.15.69:0/2165832101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e900aea90 0x7f2e900aa420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:52.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.504+0000 7f2e8effd640 1 -- 172.21.15.69:0/2165832101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e900b12c0 msgr2=0x7f2e900a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:52.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.504+0000 7f2e8effd640 1 --2- 172.21.15.69:0/2165832101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e900b12c0 0x7f2e900a83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:52.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.504+0000 7f2e8effd640 10 monclient: _finish_hunting 0 2023-10-30T20:45:52.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.504+0000 7f2e8effd640 1 monclient: found mon.noname-a 2023-10-30T20:45:52.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.504+0000 7f2e8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:52.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.504+0000 7f2e8effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:45:52.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.504+0000 7f2e8effd640 1 -- 172.21.15.69:0/2165832101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e900aa060 con 0x7f2e900ada60 2023-10-30T20:45:52.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.504+0000 7f2e8e7fc640 1 --2- 172.21.15.69:0/2165832101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e900aea90 0x7f2e900aa420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T20:45:52.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.507+0000 7f2e8effd640 1 --2- 172.21.15.69:0/2165832101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e900ada60 0x7f2e90144f20 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000c8a0 tx=0x7f2e8000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:52.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.507+0000 7f2e77fff640 10 client.?.objecter ms_handle_connect 0x7f2e900ada60 2023-10-30T20:45:52.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.507+0000 7f2e77fff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:52.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e77fff640 1 -- 172.21.15.69:0/2165832101 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80003b70 con 0x7f2e900ada60 2023-10-30T20:45:52.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:52.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:45:52.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e77fff640 10 monclient: dump: 2023-10-30T20:45:52.527 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:52.527 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:52.527 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:52.527 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:52.528 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:52.528 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:52.528 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:52.528 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:52.528 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:52.529 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:52.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e77fff640 10 monclient: _finish_auth 0 2023-10-30T20:45:52.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e77fff640 10 monclient: _check_auth_tickets 2023-10-30T20:45:52.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:52.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e77fff640 1 -- 172.21.15.69:0/2165832101 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8000e040 con 0x7f2e900ada60 2023-10-30T20:45:52.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e77fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:52.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e77fff640 1 -- 172.21.15.69:0/2165832101 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8000a7f0 con 0x7f2e900ada60 2023-10-30T20:45:52.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e95a02640 5 monclient: authenticate success, global_id 16227 2023-10-30T20:45:52.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e95a02640 10 monclient: _renew_subs 2023-10-30T20:45:52.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e95a02640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:45:52.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e95a02640 1 -- 172.21.15.69:0/2165832101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e900a8910 con 0x7f2e900ada60 2023-10-30T20:45:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e8d7fa640 4 set_mon_vals no callback set 2023-10-30T20:45:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:45:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e77fff640 10 monclient: dump: 2023-10-30T20:45:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:52.533 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e95a02640 10 client.16227.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e95a02640 10 monclient: _renew_subs 2023-10-30T20:45:52.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e95a02640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:45:52.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e95a02640 1 -- 172.21.15.69:0/2165832101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e900a8e00 con 0x7f2e900ada60 2023-10-30T20:45:52.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.510+0000 7f2e95a02640 1 librados: init done 2023-10-30T20:45:52.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.511+0000 7f2e95a02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:52.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.511+0000 7f2e95a02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:52.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.511+0000 7f2e95a02640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:45:52.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.511+0000 7f2e95a02640 1 -- 172.21.15.69:0/2165832101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e900a9000 con 0x7f2e900ada60 2023-10-30T20:45:52.765 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.764+0000 7f2e77fff640 1 -- 172.21.15.69:0/2165832101 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f2e8000a960 con 0x7f2e900ada60 2023-10-30T20:45:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.765+0000 7f2e77fff640 1 --2- 172.21.15.69:0/2165832101 >> v2:172.21.15.69:6800/1876307353 conn(0x7f2e7c06c810 0x7f2e7c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.765+0000 7f2e8e7fc640 1 --2- 172.21.15.69:0/2165832101 >> v2:172.21.15.69:6800/1876307353 conn(0x7f2e7c06c810 0x7f2e7c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.765+0000 7f2e8e7fc640 10 monclient: get_auth_request con 0x7f2e7c06c810 auth_method 0 2023-10-30T20:45:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.765+0000 7f2e77fff640 10 client.16227.objecter ms_handle_connect 0x7f2e7c06c810 2023-10-30T20:45:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.765+0000 7f2e8e7fc640 1 --2- 172.21.15.69:0/2165832101 >> v2:172.21.15.69:6800/1876307353 conn(0x7f2e7c06c810 0x7f2e7c06ecd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2e84036000 tx=0x7f2e8403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.779+0000 7f2e77fff640 1 -- 172.21.15.69:0/2165832101 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(903..903 src has 451..903) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f2e800585a0 con 0x7f2e900ada60 2023-10-30T20:45:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.779+0000 7f2e77fff640 10 client.16227.objecter ms_dispatch 0x7f2e90000b90 osd_map(903..903 src has 451..903) v4 2023-10-30T20:45:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.779+0000 7f2e77fff640 3 client.16227.objecter handle_osd_map got epochs [903,903] > 0 2023-10-30T20:45:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.779+0000 7f2e77fff640 3 client.16227.objecter handle_osd_map decoding full epoch 903 2023-10-30T20:45:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.780+0000 7f2e77fff640 20 client.16227.objecter dump_active .. 0 homeless 2023-10-30T20:45:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.780+0000 7f2e77fff640 1 -- 172.21.15.69:0/2165832101 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2e8005c640 con 0x7f2e900ada60 2023-10-30T20:45:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.780+0000 7f2e77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.780+0000 7f2e77fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:45:52.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.951+0000 7f2e95a02640 1 -- 172.21.15.69:0/2165832101 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2e900aacd0 con 0x7f2e7c06c810 2023-10-30T20:45:52.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.956+0000 7f2e77fff640 1 -- 172.21.15.69:0/2165832101 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151630 (secure 0 0 0) 0x7f2e900aacd0 con 0x7f2e7c06c810 2023-10-30T20:45:52.959 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:45:52.960 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-10-30T20:45:52.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 10 librados: watch_flush enter 2023-10-30T20:45:52.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 10 librados: watch_flush exit 2023-10-30T20:45:52.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 20 client.16227.objecter shutdown clearing up homeless session... 2023-10-30T20:45:52.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 10 client.16227.objecter successfully canceled tick 2023-10-30T20:45:52.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 1 -- 172.21.15.69:0/2165832101 >> v2:172.21.15.69:6800/1876307353 conn(0x7f2e7c06c810 msgr2=0x7f2e7c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:52.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 1 --2- 172.21.15.69:0/2165832101 >> v2:172.21.15.69:6800/1876307353 conn(0x7f2e7c06c810 0x7f2e7c06ecd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2e84036000 tx=0x7f2e8403c040 comp rx=0 tx=0).stop 2023-10-30T20:45:52.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 10 monclient: shutdown 2023-10-30T20:45:52.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:52.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 1 -- 172.21.15.69:0/2165832101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e900ada60 msgr2=0x7f2e90144f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:52.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 1 --2- 172.21.15.69:0/2165832101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e900ada60 0x7f2e90144f20 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000c8a0 tx=0x7f2e8000cd60 comp rx=0 tx=0).stop 2023-10-30T20:45:52.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 1 -- 172.21.15.69:0/2165832101 shutdown_connections 2023-10-30T20:45:52.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 1 --2- 172.21.15.69:0/2165832101 >> v2:172.21.15.69:6800/1876307353 conn(0x7f2e7c06c810 0x7f2e7c06ecd0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:52.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 1 --2- 172.21.15.69:0/2165832101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e900b12c0 0x7f2e900a83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:52.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 1 --2- 172.21.15.69:0/2165832101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e900aea90 0x7f2e900aa420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:52.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 1 --2- 172.21.15.69:0/2165832101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e900ada60 0x7f2e90144f20 unknown :-1 s=CLOSED pgs=3172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:52.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 1 -- 172.21.15.69:0/2165832101 >> 172.21.15.69:0/2165832101 conn(0x7f2e900b7120 msgr2=0x7f2e900b42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:52.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 1 -- 172.21.15.69:0/2165832101 shutdown_connections 2023-10-30T20:45:52.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 1 -- 172.21.15.69:0/2165832101 wait complete. 2023-10-30T20:45:52.965 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:52.958+0000 7f2e95a02640 1 librados: shutdown 2023-10-30T20:45:52.971 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1567,"stamp":"2023-10-30T20:45:52.503447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":161,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":545524,"kb_used_data":51772,"kb_used_omap":12,"kb_used_meta":493619,"kb_avail":754429196,"statfs":{"total":773094113280,"available":772535496704,"internally_reserved":0,"allocated":53014528,"data_stored":25957640,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12745,"internal_metadata":505466423},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":405,"apply_latency_ms":405,"commit_latency_ns":405000000,"apply_latency_ns":405000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.129016"},"pg_stats":[{"pgid":"10.16","version":"0'0","reported_seq":845,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.917971+0000","last_change":"2023-10-30T20:25:23.971023+0000","last_active":"2023-10-30T20:45:46.917971+0000","last_peered":"2023-10-30T20:45:46.917971+0000","last_clean":"2023-10-30T20:45:46.917971+0000","last_became_active":"2023-10-30T20:25:23.962078+0000","last_became_peered":"2023-10-30T20:25:23.962078+0000","last_unstale":"2023-10-30T20:45:46.917971+0000","last_undegraded":"2023-10-30T20:45:46.917971+0000","last_fullsized":"2023-10-30T20:45:46.917971+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:31.404506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":869,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.942454+0000","last_change":"2023-10-30T20:25:24.870906+0000","last_active":"2023-10-30T20:45:46.942454+0000","last_peered":"2023-10-30T20:45:46.942454+0000","last_clean":"2023-10-30T20:45:46.942454+0000","last_became_active":"2023-10-30T20:25:24.861824+0000","last_became_peered":"2023-10-30T20:25:24.861824+0000","last_unstale":"2023-10-30T20:45:46.942454+0000","last_undegraded":"2023-10-30T20:45:46.942454+0000","last_fullsized":"2023-10-30T20:45:46.942454+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:40.886429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":851,"reported_epoch":902,"state":"active+clean","last_fresh":"2023-10-30T20:45:49.656159+0000","last_change":"2023-10-30T20:25:36.960712+0000","last_active":"2023-10-30T20:45:49.656159+0000","last_peered":"2023-10-30T20:45:49.656159+0000","last_clean":"2023-10-30T20:45:49.656159+0000","last_became_active":"2023-10-30T20:25:36.951760+0000","last_became_peered":"2023-10-30T20:25:36.951760+0000","last_unstale":"2023-10-30T20:45:49.656159+0000","last_undegraded":"2023-10-30T20:45:49.656159+0000","last_fullsized":"2023-10-30T20:45:49.656159+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:17:29.613581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":866,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.937918+0000","last_change":"2023-10-30T20:25:23.364080+0000","last_active":"2023-10-30T20:45:46.937918+0000","last_peered":"2023-10-30T20:45:46.937918+0000","last_clean":"2023-10-30T20:45:46.937918+0000","last_became_active":"2023-10-30T20:25:23.356998+0000","last_became_peered":"2023-10-30T20:25:23.356998+0000","last_unstale":"2023-10-30T20:45:46.937918+0000","last_undegraded":"2023-10-30T20:45:46.937918+0000","last_fullsized":"2023-10-30T20:45:46.937918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:08:01.028636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.468387+0000","last_change":"2023-10-30T20:25:25.485385+0000","last_active":"2023-10-30T20:45:47.468387+0000","last_peered":"2023-10-30T20:45:47.468387+0000","last_clean":"2023-10-30T20:45:47.468387+0000","last_became_active":"2023-10-30T20:25:25.477142+0000","last_became_peered":"2023-10-30T20:25:25.477142+0000","last_unstale":"2023-10-30T20:45:47.468387+0000","last_undegraded":"2023-10-30T20:45:47.468387+0000","last_fullsized":"2023-10-30T20:45:47.468387+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:26:58.277816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":851,"reported_epoch":902,"state":"active+clean","last_fresh":"2023-10-30T20:45:49.660993+0000","last_change":"2023-10-30T20:25:23.846753+0000","last_active":"2023-10-30T20:45:49.660993+0000","last_peered":"2023-10-30T20:45:49.660993+0000","last_clean":"2023-10-30T20:45:49.660993+0000","last_became_active":"2023-10-30T20:25:23.840599+0000","last_became_peered":"2023-10-30T20:25:23.840599+0000","last_unstale":"2023-10-30T20:45:49.660993+0000","last_undegraded":"2023-10-30T20:45:49.660993+0000","last_fullsized":"2023-10-30T20:45:49.660993+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:46.666701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.439563+0000","last_change":"2023-10-30T20:25:28.699156+0000","last_active":"2023-10-30T20:45:47.439563+0000","last_peered":"2023-10-30T20:45:47.439563+0000","last_clean":"2023-10-30T20:45:47.439563+0000","last_became_active":"2023-10-30T20:25:28.691162+0000","last_became_peered":"2023-10-30T20:25:28.691162+0000","last_unstale":"2023-10-30T20:45:47.439563+0000","last_undegraded":"2023-10-30T20:45:47.439563+0000","last_fullsized":"2023-10-30T20:45:47.439563+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:51:37.793199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.449465+0000","last_change":"2023-10-30T20:25:36.297471+0000","last_active":"2023-10-30T20:45:47.449465+0000","last_peered":"2023-10-30T20:45:47.449465+0000","last_clean":"2023-10-30T20:45:47.449465+0000","last_became_active":"2023-10-30T20:25:36.286902+0000","last_became_peered":"2023-10-30T20:25:36.286902+0000","last_unstale":"2023-10-30T20:45:47.449465+0000","last_undegraded":"2023-10-30T20:45:47.449465+0000","last_fullsized":"2023-10-30T20:45:47.449465+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:34.822570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":856,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.586217+0000","last_change":"2023-10-30T20:25:25.388067+0000","last_active":"2023-10-30T20:45:47.586217+0000","last_peered":"2023-10-30T20:45:47.586217+0000","last_clean":"2023-10-30T20:45:47.586217+0000","last_became_active":"2023-10-30T20:25:25.367753+0000","last_became_peered":"2023-10-30T20:25:25.367753+0000","last_unstale":"2023-10-30T20:45:47.586217+0000","last_undegraded":"2023-10-30T20:45:47.586217+0000","last_fullsized":"2023-10-30T20:45:47.586217+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:29:40.135060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":854,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.941210+0000","last_change":"2023-10-30T20:25:33.376337+0000","last_active":"2023-10-30T20:45:46.941210+0000","last_peered":"2023-10-30T20:45:46.941210+0000","last_clean":"2023-10-30T20:45:46.941210+0000","last_became_active":"2023-10-30T20:25:33.362871+0000","last_became_peered":"2023-10-30T20:25:33.362871+0000","last_unstale":"2023-10-30T20:45:46.941210+0000","last_undegraded":"2023-10-30T20:45:46.941210+0000","last_fullsized":"2023-10-30T20:45:46.941210+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:57:41.166559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":855,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.894026+0000","last_change":"2023-10-30T20:25:35.098879+0000","last_active":"2023-10-30T20:45:46.894026+0000","last_peered":"2023-10-30T20:45:46.894026+0000","last_clean":"2023-10-30T20:45:46.894026+0000","last_became_active":"2023-10-30T20:25:35.047116+0000","last_became_peered":"2023-10-30T20:25:35.047116+0000","last_unstale":"2023-10-30T20:45:46.894026+0000","last_undegraded":"2023-10-30T20:45:46.894026+0000","last_fullsized":"2023-10-30T20:45:46.894026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:54:18.479663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":869,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.954888+0000","last_change":"2023-10-30T20:25:45.287217+0000","last_active":"2023-10-30T20:45:46.954888+0000","last_peered":"2023-10-30T20:45:46.954888+0000","last_clean":"2023-10-30T20:45:46.954888+0000","last_became_active":"2023-10-30T20:25:45.271565+0000","last_became_peered":"2023-10-30T20:25:45.271565+0000","last_unstale":"2023-10-30T20:45:46.954888+0000","last_undegraded":"2023-10-30T20:45:46.954888+0000","last_fullsized":"2023-10-30T20:45:46.954888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:35:16.997608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":870,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.939320+0000","last_change":"2023-10-30T20:25:28.837575+0000","last_active":"2023-10-30T20:45:46.939320+0000","last_peered":"2023-10-30T20:45:46.939320+0000","last_clean":"2023-10-30T20:45:46.939320+0000","last_became_active":"2023-10-30T20:25:28.829991+0000","last_became_peered":"2023-10-30T20:25:28.829991+0000","last_unstale":"2023-10-30T20:45:46.939320+0000","last_undegraded":"2023-10-30T20:45:46.939320+0000","last_fullsized":"2023-10-30T20:45:46.939320+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:05:54.663012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.937750+0000","last_change":"2023-10-30T20:24:04.700492+0000","last_active":"2023-10-30T20:45:46.937750+0000","last_peered":"2023-10-30T20:45:46.937750+0000","last_clean":"2023-10-30T20:45:46.937750+0000","last_became_active":"2023-10-30T20:24:04.692769+0000","last_became_peered":"2023-10-30T20:24:04.692769+0000","last_unstale":"2023-10-30T20:45:46.937750+0000","last_undegraded":"2023-10-30T20:45:46.937750+0000","last_fullsized":"2023-10-30T20:45:46.937750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:58.321494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":846,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.919523+0000","last_change":"2023-10-30T20:25:27.300642+0000","last_active":"2023-10-30T20:45:46.919523+0000","last_peered":"2023-10-30T20:45:46.919523+0000","last_clean":"2023-10-30T20:45:46.919523+0000","last_became_active":"2023-10-30T20:25:27.291944+0000","last_became_peered":"2023-10-30T20:25:27.291944+0000","last_unstale":"2023-10-30T20:45:46.919523+0000","last_undegraded":"2023-10-30T20:45:46.919523+0000","last_fullsized":"2023-10-30T20:45:46.919523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:32:41.544931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":872,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.908991+0000","last_change":"2023-10-30T20:24:04.738314+0000","last_active":"2023-10-30T20:45:46.908991+0000","last_peered":"2023-10-30T20:45:46.908991+0000","last_clean":"2023-10-30T20:45:46.908991+0000","last_became_active":"2023-10-30T20:24:04.712006+0000","last_became_peered":"2023-10-30T20:24:04.712006+0000","last_unstale":"2023-10-30T20:45:46.908991+0000","last_undegraded":"2023-10-30T20:45:46.908991+0000","last_fullsized":"2023-10-30T20:45:46.908991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:19:34.828603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":847,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.928609+0000","last_change":"2023-10-30T20:25:39.862114+0000","last_active":"2023-10-30T20:45:46.928609+0000","last_peered":"2023-10-30T20:45:46.928609+0000","last_clean":"2023-10-30T20:45:46.928609+0000","last_became_active":"2023-10-30T20:25:39.852846+0000","last_became_peered":"2023-10-30T20:25:39.852846+0000","last_unstale":"2023-10-30T20:45:46.928609+0000","last_undegraded":"2023-10-30T20:45:46.928609+0000","last_fullsized":"2023-10-30T20:45:46.928609+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:00:12.439335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.951807+0000","last_change":"2023-10-30T20:24:04.315228+0000","last_active":"2023-10-30T20:45:46.951807+0000","last_peered":"2023-10-30T20:45:46.951807+0000","last_clean":"2023-10-30T20:45:46.951807+0000","last_became_active":"2023-10-30T20:24:04.310065+0000","last_became_peered":"2023-10-30T20:24:04.310065+0000","last_unstale":"2023-10-30T20:45:46.951807+0000","last_undegraded":"2023-10-30T20:45:46.951807+0000","last_fullsized":"2023-10-30T20:45:46.951807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:09:06.876832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":856,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.895586+0000","last_change":"2023-10-30T20:25:42.330893+0000","last_active":"2023-10-30T20:45:46.895586+0000","last_peered":"2023-10-30T20:45:46.895586+0000","last_clean":"2023-10-30T20:45:46.895586+0000","last_became_active":"2023-10-30T20:25:42.322505+0000","last_became_peered":"2023-10-30T20:25:42.322505+0000","last_unstale":"2023-10-30T20:45:46.895586+0000","last_undegraded":"2023-10-30T20:45:46.895586+0000","last_fullsized":"2023-10-30T20:45:46.895586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:04.210219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":860,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.609161+0000","last_change":"2023-10-30T20:24:04.552803+0000","last_active":"2023-10-30T20:45:47.609161+0000","last_peered":"2023-10-30T20:45:47.609161+0000","last_clean":"2023-10-30T20:45:47.609161+0000","last_became_active":"2023-10-30T20:24:04.525673+0000","last_became_peered":"2023-10-30T20:24:04.525673+0000","last_unstale":"2023-10-30T20:45:47.609161+0000","last_undegraded":"2023-10-30T20:45:47.609161+0000","last_fullsized":"2023-10-30T20:45:47.609161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:18.627320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":856,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.906498+0000","last_change":"2023-10-30T20:25:33.256701+0000","last_active":"2023-10-30T20:45:46.906498+0000","last_peered":"2023-10-30T20:45:46.906498+0000","last_clean":"2023-10-30T20:45:46.906498+0000","last_became_active":"2023-10-30T20:25:33.242618+0000","last_became_peered":"2023-10-30T20:25:33.242618+0000","last_unstale":"2023-10-30T20:45:46.906498+0000","last_undegraded":"2023-10-30T20:45:46.906498+0000","last_fullsized":"2023-10-30T20:45:46.906498+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:56.435103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.910532+0000","last_change":"2023-10-30T20:24:04.525394+0000","last_active":"2023-10-30T20:45:46.910532+0000","last_peered":"2023-10-30T20:45:46.910532+0000","last_clean":"2023-10-30T20:45:46.910532+0000","last_became_active":"2023-10-30T20:24:04.512341+0000","last_became_peered":"2023-10-30T20:24:04.512341+0000","last_unstale":"2023-10-30T20:45:46.910532+0000","last_undegraded":"2023-10-30T20:45:46.910532+0000","last_fullsized":"2023-10-30T20:45:46.910532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:00:36.886515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":847,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.889757+0000","last_change":"2023-10-30T20:25:25.171687+0000","last_active":"2023-10-30T20:45:46.889757+0000","last_peered":"2023-10-30T20:45:46.889757+0000","last_clean":"2023-10-30T20:45:46.889757+0000","last_became_active":"2023-10-30T20:25:25.151444+0000","last_became_peered":"2023-10-30T20:25:25.151444+0000","last_unstale":"2023-10-30T20:45:46.889757+0000","last_undegraded":"2023-10-30T20:45:46.889757+0000","last_fullsized":"2023-10-30T20:45:46.889757+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:25:32.523488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":872,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-10-30T20:45:48.444468+0000","last_change":"2023-10-30T20:24:05.504482+0000","last_active":"2023-10-30T20:45:48.444468+0000","last_peered":"2023-10-30T20:45:48.444468+0000","last_clean":"2023-10-30T20:45:48.444468+0000","last_became_active":"2023-10-30T20:24:05.475394+0000","last_became_peered":"2023-10-30T20:24:05.475394+0000","last_unstale":"2023-10-30T20:45:48.444468+0000","last_undegraded":"2023-10-30T20:45:48.444468+0000","last_fullsized":"2023-10-30T20:45:48.444468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:21:02.335096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":855,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.530050+0000","last_change":"2023-10-30T20:25:23.490954+0000","last_active":"2023-10-30T20:45:47.530050+0000","last_peered":"2023-10-30T20:45:47.530050+0000","last_clean":"2023-10-30T20:45:47.530050+0000","last_became_active":"2023-10-30T20:25:23.473157+0000","last_became_peered":"2023-10-30T20:25:23.473157+0000","last_unstale":"2023-10-30T20:45:47.530050+0000","last_undegraded":"2023-10-30T20:45:47.530050+0000","last_fullsized":"2023-10-30T20:45:47.530050+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:47:35.168516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.896471+0000","last_change":"2023-10-30T20:24:04.284202+0000","last_active":"2023-10-30T20:45:46.896471+0000","last_peered":"2023-10-30T20:45:46.896471+0000","last_clean":"2023-10-30T20:45:46.896471+0000","last_became_active":"2023-10-30T20:24:04.254051+0000","last_became_peered":"2023-10-30T20:24:04.254051+0000","last_unstale":"2023-10-30T20:45:46.896471+0000","last_undegraded":"2023-10-30T20:45:46.896471+0000","last_fullsized":"2023-10-30T20:45:46.896471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:14:06.705421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":878,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.465358+0000","last_change":"2023-10-30T20:25:36.268959+0000","last_active":"2023-10-30T20:45:47.465358+0000","last_peered":"2023-10-30T20:45:47.465358+0000","last_clean":"2023-10-30T20:45:47.465358+0000","last_became_active":"2023-10-30T20:25:36.260481+0000","last_became_peered":"2023-10-30T20:25:36.260481+0000","last_unstale":"2023-10-30T20:45:47.465358+0000","last_undegraded":"2023-10-30T20:45:47.465358+0000","last_fullsized":"2023-10-30T20:45:47.465358+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:06:50.808567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.917896+0000","last_change":"2023-10-30T20:24:04.676966+0000","last_active":"2023-10-30T20:45:46.917896+0000","last_peered":"2023-10-30T20:45:46.917896+0000","last_clean":"2023-10-30T20:45:46.917896+0000","last_became_active":"2023-10-30T20:24:04.671790+0000","last_became_peered":"2023-10-30T20:24:04.671790+0000","last_unstale":"2023-10-30T20:45:46.917896+0000","last_undegraded":"2023-10-30T20:45:46.917896+0000","last_fullsized":"2023-10-30T20:45:46.917896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:56.600896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":989,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.441313+0000","last_change":"2023-10-30T20:23:57.127569+0000","last_active":"2023-10-30T20:45:47.441313+0000","last_peered":"2023-10-30T20:45:47.441313+0000","last_clean":"2023-10-30T20:45:47.441313+0000","last_became_active":"2023-10-30T20:23:57.098872+0000","last_became_peered":"2023-10-30T20:23:57.098872+0000","last_unstale":"2023-10-30T20:45:47.441313+0000","last_undegraded":"2023-10-30T20:45:47.441313+0000","last_fullsized":"2023-10-30T20:45:47.441313+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:23:54.534733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:23:54.534733+0000","last_clean_scrub_stamp":"2023-10-30T20:23:54.534733+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:26:29.039781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"45'7","reported_seq":860,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.892575+0000","last_change":"2023-10-30T20:25:25.378575+0000","last_active":"2023-10-30T20:45:46.892575+0000","last_peered":"2023-10-30T20:45:46.892575+0000","last_clean":"2023-10-30T20:45:46.892575+0000","last_became_active":"2023-10-30T20:25:25.362636+0000","last_became_peered":"2023-10-30T20:25:25.362636+0000","last_unstale":"2023-10-30T20:45:46.892575+0000","last_undegraded":"2023-10-30T20:45:46.892575+0000","last_fullsized":"2023-10-30T20:45:46.892575+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:34.822570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":857,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.915351+0000","last_change":"2023-10-30T20:25:23.612993+0000","last_active":"2023-10-30T20:45:46.915351+0000","last_peered":"2023-10-30T20:45:46.915351+0000","last_clean":"2023-10-30T20:45:46.915351+0000","last_became_active":"2023-10-30T20:25:23.607599+0000","last_became_peered":"2023-10-30T20:25:23.607599+0000","last_unstale":"2023-10-30T20:45:46.915351+0000","last_undegraded":"2023-10-30T20:45:46.915351+0000","last_fullsized":"2023-10-30T20:45:46.915351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:25:35.997128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":867,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.899233+0000","last_change":"2023-10-30T20:25:28.335295+0000","last_active":"2023-10-30T20:45:46.899233+0000","last_peered":"2023-10-30T20:45:46.899233+0000","last_clean":"2023-10-30T20:45:46.899233+0000","last_became_active":"2023-10-30T20:25:28.326296+0000","last_became_peered":"2023-10-30T20:25:28.326296+0000","last_unstale":"2023-10-30T20:45:46.899233+0000","last_undegraded":"2023-10-30T20:45:46.899233+0000","last_fullsized":"2023-10-30T20:45:46.899233+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:41:04.620931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":855,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.483979+0000","last_change":"2023-10-30T20:25:20.677355+0000","last_active":"2023-10-30T20:45:47.483979+0000","last_peered":"2023-10-30T20:45:47.483979+0000","last_clean":"2023-10-30T20:45:47.483979+0000","last_became_active":"2023-10-30T20:25:20.671965+0000","last_became_peered":"2023-10-30T20:25:20.671965+0000","last_unstale":"2023-10-30T20:45:47.483979+0000","last_undegraded":"2023-10-30T20:45:47.483979+0000","last_fullsized":"2023-10-30T20:45:47.483979+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:55:35.886983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":867,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.936385+0000","last_change":"2023-10-30T20:25:28.987986+0000","last_active":"2023-10-30T20:45:46.936385+0000","last_peered":"2023-10-30T20:45:46.936385+0000","last_clean":"2023-10-30T20:45:46.936385+0000","last_became_active":"2023-10-30T20:25:28.979668+0000","last_became_peered":"2023-10-30T20:25:28.979668+0000","last_unstale":"2023-10-30T20:45:46.936385+0000","last_undegraded":"2023-10-30T20:45:46.936385+0000","last_fullsized":"2023-10-30T20:45:46.936385+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:22:41.686629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":867,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.902220+0000","last_change":"2023-10-30T20:25:23.069394+0000","last_active":"2023-10-30T20:45:46.902220+0000","last_peered":"2023-10-30T20:45:46.902220+0000","last_clean":"2023-10-30T20:45:46.902220+0000","last_became_active":"2023-10-30T20:25:23.058176+0000","last_became_peered":"2023-10-30T20:25:23.058176+0000","last_unstale":"2023-10-30T20:45:46.902220+0000","last_undegraded":"2023-10-30T20:45:46.902220+0000","last_fullsized":"2023-10-30T20:45:46.902220+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:33:56.491589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":870,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.940844+0000","last_change":"2023-10-30T20:25:28.823327+0000","last_active":"2023-10-30T20:45:46.940844+0000","last_peered":"2023-10-30T20:45:46.940844+0000","last_clean":"2023-10-30T20:45:46.940844+0000","last_became_active":"2023-10-30T20:25:28.798013+0000","last_became_peered":"2023-10-30T20:25:28.798013+0000","last_unstale":"2023-10-30T20:45:46.940844+0000","last_undegraded":"2023-10-30T20:45:46.940844+0000","last_fullsized":"2023-10-30T20:45:46.940844+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:35:37.042427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":870,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.931482+0000","last_change":"2023-10-30T20:25:23.723655+0000","last_active":"2023-10-30T20:45:46.931482+0000","last_peered":"2023-10-30T20:45:46.931482+0000","last_clean":"2023-10-30T20:45:46.931482+0000","last_became_active":"2023-10-30T20:25:23.709112+0000","last_became_peered":"2023-10-30T20:25:23.709112+0000","last_unstale":"2023-10-30T20:45:46.931482+0000","last_undegraded":"2023-10-30T20:45:46.931482+0000","last_fullsized":"2023-10-30T20:45:46.931482+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:06:46.124278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":851,"reported_epoch":902,"state":"active+clean","last_fresh":"2023-10-30T20:45:49.650763+0000","last_change":"2023-10-30T20:25:28.781780+0000","last_active":"2023-10-30T20:45:49.650763+0000","last_peered":"2023-10-30T20:45:49.650763+0000","last_clean":"2023-10-30T20:45:49.650763+0000","last_became_active":"2023-10-30T20:25:28.762953+0000","last_became_peered":"2023-10-30T20:25:28.762953+0000","last_unstale":"2023-10-30T20:45:49.650763+0000","last_undegraded":"2023-10-30T20:45:49.650763+0000","last_fullsized":"2023-10-30T20:45:49.650763+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:09.371593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":866,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.918142+0000","last_change":"2023-10-30T20:25:21.917941+0000","last_active":"2023-10-30T20:45:46.918142+0000","last_peered":"2023-10-30T20:45:46.918142+0000","last_clean":"2023-10-30T20:45:46.918142+0000","last_became_active":"2023-10-30T20:25:21.911569+0000","last_became_peered":"2023-10-30T20:25:21.911569+0000","last_unstale":"2023-10-30T20:45:46.918142+0000","last_undegraded":"2023-10-30T20:45:46.918142+0000","last_fullsized":"2023-10-30T20:45:46.918142+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:32:26.723949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":873,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T20:45:46.897611+0000","last_change":"2023-10-30T20:29:14.877146+0000","last_active":"2023-10-30T20:45:46.897611+0000","last_peered":"2023-10-30T20:45:46.897611+0000","last_clean":"2023-10-30T20:45:46.897611+0000","last_became_active":"2023-10-30T20:29:14.860294+0000","last_became_peered":"2023-10-30T20:29:14.860294+0000","last_unstale":"2023-10-30T20:45:46.897611+0000","last_undegraded":"2023-10-30T20:45:46.897611+0000","last_fullsized":"2023-10-30T20:45:46.897611+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:31:25.822566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T20:45:47.460543+0000","last_change":"2023-10-30T20:25:29.021159+0000","last_active":"2023-10-30T20:45:47.460543+0000","last_peered":"2023-10-30T20:45:47.460543+0000","last_clean":"2023-10-30T20:45:47.460543+0000","last_became_active":"2023-10-30T20:25:29.004167+0000","last_became_peered":"2023-10-30T20:25:29.004167+0000","last_unstale":"2023-10-30T20:45:47.460543+0000","last_undegraded":"2023-10-30T20:45:47.460543+0000","last_fullsized":"2023-10-30T20:45:47.460543+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T08:00:40.390424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477001,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59904,"kb_used_data":6516,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311936,"statfs":{"total":96636764160,"available":96575422464,"internally_reserved":0,"allocated":6672384,"data_stored":3197595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":25.940000000000001,"5min":18.199999999999999,"15min":16.863},"min":{"1min":5.1040000000000001,"5min":5.0529999999999999,"15min":3.9740000000000002},"max":{"1min":156.071,"5min":222.095,"15min":244.999},"last":8.1519999999999992},{"interface":"front","average":{"1min":32.825000000000003,"5min":20.239000000000001,"15min":18.428000000000001},"min":{"1min":5.1020000000000003,"5min":4.9279999999999999,"15min":4.2539999999999996},"max":{"1min":219.797,"5min":219.797,"15min":219.797},"last":6.5129999999999999}]},{"osd":1,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":25.375,"5min":20.100999999999999,"15min":20.373000000000001},"min":{"1min":9.6869999999999994,"5min":5.6500000000000004,"15min":4.5940000000000003},"max":{"1min":143.87200000000001,"5min":231.85499999999999,"15min":256.815},"last":10.204000000000001},{"interface":"front","average":{"1min":22.506,"5min":19.602,"15min":22.550999999999998},"min":{"1min":7.298,"5min":5.7320000000000002,"15min":5.726},"max":{"1min":122.70399999999999,"5min":237.88300000000001,"15min":424.57999999999998},"last":14.696999999999999}]},{"osd":2,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":34.981999999999999,"5min":24.867999999999999,"15min":24.663},"min":{"1min":9.7789999999999999,"5min":6.1539999999999999,"15min":6.1539999999999999},"max":{"1min":267.81099999999998,"5min":267.81099999999998,"15min":835.96799999999996},"last":10.823},{"interface":"front","average":{"1min":37.021999999999998,"5min":26.030999999999999,"15min":23.010000000000002},"min":{"1min":10.771000000000001,"5min":10.173,"15min":8.2479999999999993},"max":{"1min":244.50399999999999,"5min":247.178,"15min":261.50900000000001},"last":13.718999999999999}]},{"osd":4,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":29.966000000000001,"5min":27.501000000000001,"15min":23.704999999999998},"min":{"1min":6.8390000000000004,"5min":6.625,"15min":5.5190000000000001},"max":{"1min":196.904,"5min":247.904,"15min":247.904},"last":9.1969999999999992},{"interface":"front","average":{"1min":29.170999999999999,"5min":25.465,"15min":24.071000000000002},"min":{"1min":7.9900000000000002,"5min":6.8620000000000001,"15min":6.3840000000000003},"max":{"1min":210.88200000000001,"5min":235.64099999999999,"15min":235.64099999999999},"last":15.352}]},{"osd":5,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":35.137999999999998,"5min":26.385999999999999,"15min":25.082999999999998},"min":{"1min":8.4100000000000001,"5min":6.726,"15min":6.726},"max":{"1min":198.76300000000001,"5min":233.28999999999999,"15min":233.28999999999999},"last":17.306999999999999},{"interface":"front","average":{"1min":46.277000000000001,"5min":31.02,"15min":24.657},"min":{"1min":11.351000000000001,"5min":10.645,"15min":7.883},"max":{"1min":227.28299999999999,"5min":244.23500000000001,"15min":244.23500000000001},"last":18.056999999999999}]},{"osd":6,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":41.406999999999996,"5min":30.888000000000002,"15min":27.672999999999998},"min":{"1min":11.547000000000001,"5min":9.1989999999999998,"15min":9.1989999999999998},"max":{"1min":269.15600000000001,"5min":269.15600000000001,"15min":269.15600000000001},"last":20.324999999999999},{"interface":"front","average":{"1min":32.853999999999999,"5min":24.702999999999999,"15min":22.295999999999999},"min":{"1min":6.093,"5min":6.093,"15min":6.093},"max":{"1min":262.53500000000003,"5min":262.53500000000003,"15min":262.53500000000003},"last":13.135}]},{"osd":7,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":29.765999999999998,"5min":27.036000000000001,"15min":24.699000000000002},"min":{"1min":9.7669999999999995,"5min":8.5570000000000004,"15min":7.3799999999999999},"max":{"1min":222.32400000000001,"5min":264.48000000000002,"15min":264.48000000000002},"last":16.132000000000001},{"interface":"front","average":{"1min":31.981999999999999,"5min":27.591000000000001,"15min":26.734999999999999},"min":{"1min":9.2919999999999998,"5min":8.9700000000000006,"15min":8.9700000000000006},"max":{"1min":233.99700000000001,"5min":245.042,"15min":245.042},"last":18.952000000000002}]}]},{"osd":1,"up_from":16,"seq":68719477001,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64036,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94307804,"statfs":{"total":96636764160,"available":96571191296,"internally_reserved":0,"allocated":6635520,"data_stored":3528420,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.762,"5min":15.228999999999999,"15min":16.553999999999998},"min":{"1min":4.1130000000000004,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":98.356999999999999,"5min":244.65000000000001,"15min":244.65000000000001},"last":4.8230000000000004},{"interface":"front","average":{"1min":11.536,"5min":12.617000000000001,"15min":18.32},"min":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":3.847},"max":{"1min":39.68,"5min":65.853999999999999,"15min":407.38900000000001},"last":9.5150000000000006}]},{"osd":2,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":18.402000000000001,"5min":16.823,"15min":19.931000000000001},"min":{"1min":7.5389999999999997,"5min":7.367,"15min":4.1500000000000004},"max":{"1min":80.688000000000002,"5min":113.649,"15min":297.26499999999999},"last":13.631},{"interface":"front","average":{"1min":23.346,"5min":19.442,"15min":20.483000000000001},"min":{"1min":9.8040000000000003,"5min":8.5690000000000008,"15min":6.5469999999999997},"max":{"1min":148.03999999999999,"5min":148.03999999999999,"15min":167.12100000000001},"last":14.256}]},{"osd":3,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":17.946000000000002,"5min":19.791,"15min":21.096},"min":{"1min":8.8569999999999993,"5min":7.8029999999999999,"15min":6.5990000000000002},"max":{"1min":84.090999999999994,"5min":150.166,"15min":213.971},"last":15.526},{"interface":"front","average":{"1min":19.449000000000002,"5min":22.609000000000002,"15min":24.210999999999999},"min":{"1min":12.093999999999999,"5min":9.8409999999999993,"15min":8.1449999999999996},"max":{"1min":74.275999999999996,"5min":146.15899999999999,"15min":223.49199999999999},"last":16.167000000000002}]},{"osd":4,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":13.090999999999999,"5min":20.369,"15min":21.076000000000001},"min":{"1min":8.1349999999999998,"5min":6.8659999999999997,"15min":5.9809999999999999},"max":{"1min":30.533000000000001,"5min":200.98400000000001,"15min":215.435},"last":10.723000000000001},{"interface":"front","average":{"1min":17.289000000000001,"5min":23.798999999999999,"15min":25.364000000000001},"min":{"1min":12.874000000000001,"5min":11.516999999999999,"15min":8.9939999999999998},"max":{"1min":26.600000000000001,"5min":192.69200000000001,"15min":197.34},"last":17.629999999999999}]},{"osd":5,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":13.349,"5min":19.747,"15min":19.356000000000002},"min":{"1min":7.4020000000000001,"5min":6.4580000000000002,"15min":5.1390000000000002},"max":{"1min":62.798000000000002,"5min":209.35499999999999,"15min":241.798},"last":7.5170000000000003},{"interface":"front","average":{"1min":17.783000000000001,"5min":20.292000000000002,"15min":21.498000000000001},"min":{"1min":9.5739999999999998,"5min":6.6230000000000002,"15min":6.6230000000000002},"max":{"1min":68.882000000000005,"5min":118.65600000000001,"15min":248.124},"last":11.372}]},{"osd":6,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.948,"5min":20.902000000000001,"15min":23.236999999999998},"min":{"1min":10.205,"5min":7.7220000000000004,"15min":6.9880000000000004},"max":{"1min":37.037999999999997,"5min":204.71899999999999,"15min":406.72399999999999},"last":12.73},{"interface":"front","average":{"1min":28.690999999999999,"5min":25.748000000000001,"15min":25.416},"min":{"1min":11.704000000000001,"5min":10.397,"15min":8.4749999999999996},"max":{"1min":179.98099999999999,"5min":179.98099999999999,"15min":179.98099999999999},"last":18.324000000000002}]},{"osd":7,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":14.643000000000001,"5min":18.478999999999999,"15min":23.765000000000001},"min":{"1min":6.0410000000000004,"5min":6.0410000000000004,"15min":6.0110000000000001},"max":{"1min":58.841999999999999,"5min":188.792,"15min":249.88300000000001},"last":8.7400000000000002},{"interface":"front","average":{"1min":19.361000000000001,"5min":21.233000000000001,"15min":25.628},"min":{"1min":12.742000000000001,"5min":8.8559999999999999,"15min":7.1360000000000001},"max":{"1min":59.718000000000004,"5min":134.89400000000001,"15min":275.36700000000002},"last":14.907}]}]},{"osd":0,"up_from":16,"seq":68719477002,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59832,"kb_used_data":6448,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312008,"statfs":{"total":96636764160,"available":96575496192,"internally_reserved":0,"allocated":6602752,"data_stored":3130585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":56,"apply_latency_ms":56,"commit_latency_ns":56000000,"apply_latency_ns":56000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":14.162000000000001,"5min":13.208,"15min":18.428000000000001},"min":{"1min":5.508,"5min":4.1740000000000004,"15min":4.0819999999999999},"max":{"1min":64.745999999999995,"5min":145.857,"15min":341.85300000000001},"last":6.968},{"interface":"front","average":{"1min":15.507,"5min":16.523,"15min":18.965},"min":{"1min":5.9400000000000004,"5min":5.798,"15min":5.1669999999999998},"max":{"1min":71.578000000000003,"5min":196.148,"15min":243.62100000000001},"last":21.934999999999999}]},{"osd":2,"last update":"Mon Oct 30 20:44:59 2023","interfaces":[{"interface":"back","average":{"1min":18.457000000000001,"5min":19.640999999999998,"15min":31.206},"min":{"1min":6.5940000000000003,"5min":4.6449999999999996,"15min":4.6449999999999996},"max":{"1min":68.856999999999999,"5min":199.71100000000001,"15min":580.43100000000004},"last":6.0309999999999997},{"interface":"front","average":{"1min":15.906000000000001,"5min":18.309000000000001,"15min":31.786999999999999},"min":{"1min":6.6920000000000002,"5min":6.6920000000000002,"15min":6.1420000000000003},"max":{"1min":55.250999999999998,"5min":197.125,"15min":1090.171},"last":15.888}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.609,"5min":16.076000000000001,"15min":16.977},"min":{"1min":6.2199999999999998,"5min":5.298,"15min":5.298},"max":{"1min":76.861000000000004,"5min":148.31299999999999,"15min":167.607},"last":8.2200000000000006},{"interface":"front","average":{"1min":30.317,"5min":25.641999999999999,"15min":24.917999999999999},"min":{"1min":7.1719999999999997,"5min":7.1719999999999997,"15min":6.4370000000000003},"max":{"1min":290.19600000000003,"5min":290.19600000000003,"15min":318.245},"last":23.213999999999999}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":24.504000000000001,"5min":20.890000000000001,"15min":20.295000000000002},"min":{"1min":7.234,"5min":7.1059999999999999,"15min":5.8170000000000002},"max":{"1min":116.98399999999999,"5min":174.67699999999999,"15min":178.642},"last":11.069000000000001},{"interface":"front","average":{"1min":30.776,"5min":28.367999999999999,"15min":27.843},"min":{"1min":8.3770000000000007,"5min":8.1010000000000009,"15min":6.2960000000000003},"max":{"1min":153.124,"5min":219.14699999999999,"15min":242.06800000000001},"last":33.905999999999999}]},{"osd":5,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.25,"5min":23.315999999999999,"15min":22.43},"min":{"1min":9.3379999999999992,"5min":8.3480000000000008,"15min":6.8099999999999996},"max":{"1min":68.546999999999997,"5min":170.024,"15min":170.024},"last":26.283999999999999},{"interface":"front","average":{"1min":25.843,"5min":23.294,"15min":22.381},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.8280000000000003},"max":{"1min":138.041,"5min":177.41800000000001,"15min":177.41800000000001},"last":25.364999999999998}]},{"osd":6,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":27.763999999999999,"5min":29.074999999999999,"15min":30.167000000000002},"min":{"1min":10.817,"5min":8.7599999999999998,"15min":8.2590000000000003},"max":{"1min":151.73500000000001,"5min":189.21899999999999,"15min":330.37799999999999},"last":34.863999999999997},{"interface":"front","average":{"1min":19.260000000000002,"5min":20.901,"15min":24.004000000000001},"min":{"1min":6.8410000000000002,"5min":6.8410000000000002,"15min":6.8410000000000002},"max":{"1min":73.170000000000002,"5min":169.447,"15min":201.50299999999999},"last":28.603999999999999}]},{"osd":7,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":20.75,"5min":23.949999999999999,"15min":27.219999999999999},"min":{"1min":11.298,"5min":10.81,"15min":9.6820000000000004},"max":{"1min":70.683000000000007,"5min":175.667,"15min":268.18900000000002},"last":30.404},{"interface":"front","average":{"1min":21.437999999999999,"5min":21.204999999999998,"15min":25.626000000000001},"min":{"1min":8.8689999999999998,"5min":8.8689999999999998,"15min":7.3559999999999999},"max":{"1min":77.927000000000007,"5min":173.441,"15min":228.495},"last":27.532}]}]},{"osd":6,"up_from":15,"seq":64424509706,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59860,"kb_used_data":6468,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311980,"statfs":{"total":96636764160,"available":96575467520,"internally_reserved":0,"allocated":6623232,"data_stored":3142532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:49 2023","interfaces":[{"interface":"back","average":{"1min":15.789,"5min":12.747,"15min":13.856999999999999},"min":{"1min":4.3819999999999997,"5min":4.0179999999999998,"15min":3.7080000000000002},"max":{"1min":66.393000000000001,"5min":73.316000000000003,"15min":151.00299999999999},"last":8.8260000000000005},{"interface":"front","average":{"1min":10.926,"5min":11.757,"15min":15.211},"min":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"max":{"1min":53.823999999999998,"5min":75.060000000000002,"15min":146.49299999999999},"last":10.262}]},{"osd":1,"last update":"Mon Oct 30 20:45:49 2023","interfaces":[{"interface":"back","average":{"1min":14.625,"5min":16.163,"15min":19.315999999999999},"min":{"1min":5.8049999999999997,"5min":5.4379999999999997,"15min":4.9199999999999999},"max":{"1min":74.001000000000005,"5min":117.193,"15min":218.78100000000001},"last":13.871},{"interface":"front","average":{"1min":10.275,"5min":13.943,"15min":15.449},"min":{"1min":5.7229999999999999,"5min":5.1379999999999999,"15min":5.0149999999999997},"max":{"1min":27.335999999999999,"5min":128.50200000000001,"15min":155.27199999999999},"last":6.3280000000000003}]},{"osd":2,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":19.477,"5min":16.728000000000002,"15min":21.635000000000002},"min":{"1min":7.9080000000000004,"5min":6.1870000000000003,"15min":6.1870000000000003},"max":{"1min":67.552999999999997,"5min":122.307,"15min":222.303},"last":7.9349999999999996},{"interface":"front","average":{"1min":16.923999999999999,"5min":18.561,"15min":26.841000000000001},"min":{"1min":7.9580000000000002,"5min":7.9580000000000002,"15min":6.1820000000000004},"max":{"1min":46.755000000000003,"5min":98.781000000000006,"15min":438.19200000000001},"last":9.5990000000000002}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.385000000000002,"5min":15.81,"15min":17.850000000000001},"min":{"1min":6.4859999999999998,"5min":6.2050000000000001,"15min":6.0179999999999998},"max":{"1min":136.16800000000001,"5min":136.16800000000001,"15min":217.99000000000001},"last":105.062},{"interface":"front","average":{"1min":30.402000000000001,"5min":20.652999999999999,"15min":21.940000000000001},"min":{"1min":6.3890000000000002,"5min":6.3890000000000002,"15min":6.0010000000000003},"max":{"1min":270.25599999999997,"5min":270.25599999999997,"15min":270.25599999999997},"last":153.32900000000001}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.241,"5min":21.978000000000002,"15min":23.396000000000001},"min":{"1min":11.669,"5min":8.1539999999999999,"15min":7.5800000000000001},"max":{"1min":69.150000000000006,"5min":168.38800000000001,"15min":190.29599999999999},"last":12.670999999999999},{"interface":"front","average":{"1min":14.085000000000001,"5min":20.285,"15min":22.73},"min":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"max":{"1min":48.207000000000001,"5min":163.41800000000001,"15min":189.59399999999999},"last":10.898999999999999}]},{"osd":5,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":21.626999999999999,"5min":20.792999999999999,"15min":22.007999999999999},"min":{"1min":9.1430000000000007,"5min":8.9939999999999998,"15min":8.9930000000000003},"max":{"1min":68.114000000000004,"5min":133.73400000000001,"15min":138.38800000000001},"last":14.929},{"interface":"front","average":{"1min":25.091000000000001,"5min":24.527000000000001,"15min":25.061},"min":{"1min":12.673999999999999,"5min":11.9,"15min":10.571},"max":{"1min":68.138999999999996,"5min":150.35499999999999,"15min":152.40199999999999},"last":28.832000000000001}]},{"osd":7,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":17.923999999999999,"5min":18.893999999999998,"15min":23.824999999999999},"min":{"1min":8.8140000000000001,"5min":7.1699999999999999,"15min":6.226},"max":{"1min":64.828999999999994,"5min":98.503,"15min":253.16900000000001},"last":39.084000000000003},{"interface":"front","average":{"1min":19.021000000000001,"5min":21.308,"15min":20.143000000000001},"min":{"1min":7.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"max":{"1min":75.846000000000004,"5min":174.071,"15min":221.28399999999999},"last":17.754000000000001}]}]},{"osd":5,"up_from":15,"seq":64424509703,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77920,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293920,"statfs":{"total":96636764160,"available":96556974080,"internally_reserved":0,"allocated":6635520,"data_stored":3528420,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":17.625,"5min":18.785,"15min":16.260999999999999},"min":{"1min":5.2329999999999997,"5min":3.8359999999999999,"15min":3.8359999999999999},"max":{"1min":85.513999999999996,"5min":220.148,"15min":220.148},"last":10.542999999999999},{"interface":"front","average":{"1min":27.861000000000001,"5min":23.675000000000001,"15min":20.949000000000002},"min":{"1min":8.6039999999999992,"5min":5.6059999999999999,"15min":5.6059999999999999},"max":{"1min":145.43000000000001,"5min":177.91,"15min":177.91},"last":14.894}]},{"osd":1,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":20.715,"5min":16.585000000000001,"15min":20.353999999999999},"min":{"1min":6.3040000000000003,"5min":4.3010000000000002,"15min":4.3010000000000002},"max":{"1min":71.460999999999999,"5min":74.081000000000003,"15min":293.75400000000002},"last":9.6920000000000002},{"interface":"front","average":{"1min":20.943999999999999,"5min":16.175000000000001,"15min":18.327999999999999},"min":{"1min":5.46,"5min":4.8949999999999996,"15min":4.8949999999999996},"max":{"1min":80.397000000000006,"5min":80.397000000000006,"15min":210.797},"last":12.271000000000001}]},{"osd":2,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":31.280000000000001,"5min":23.431999999999999,"15min":25.231000000000002},"min":{"1min":6.2480000000000002,"5min":5.0209999999999999,"15min":5.0209999999999999},"max":{"1min":105.449,"5min":143.273,"15min":519.68799999999999},"last":13.766},{"interface":"front","average":{"1min":23.294,"5min":20.007000000000001,"15min":22.029},"min":{"1min":5.6479999999999997,"5min":5.6479999999999997,"15min":5.226},"max":{"1min":82.042000000000002,"5min":154.083,"15min":218.59899999999999},"last":6.0469999999999997}]},{"osd":3,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":41.817,"5min":31.161999999999999,"15min":26.263000000000002},"min":{"1min":9.9900000000000002,"5min":9.9900000000000002,"15min":7.6150000000000002},"max":{"1min":269.52999999999997,"5min":269.52999999999997,"15min":269.52999999999997},"last":16.454999999999998},{"interface":"front","average":{"1min":24.488,"5min":22.629000000000001,"15min":22.704000000000001},"min":{"1min":7.4480000000000004,"5min":6.218,"15min":5.9119999999999999},"max":{"1min":178.78700000000001,"5min":178.78700000000001,"15min":249.797},"last":6.7670000000000003}]},{"osd":4,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":21.725999999999999,"5min":26.437999999999999,"15min":25.25},"min":{"1min":8.1080000000000005,"5min":7.1580000000000004,"15min":6.6100000000000003},"max":{"1min":78.555999999999997,"5min":215.97999999999999,"15min":291.92599999999999},"last":8.1080000000000005},{"interface":"front","average":{"1min":26.097000000000001,"5min":33.600000000000001,"15min":28.585000000000001},"min":{"1min":11.895,"5min":8.0050000000000008,"15min":6.9550000000000001},"max":{"1min":70.290999999999997,"5min":324.31,"15min":324.31},"last":13.007999999999999}]},{"osd":6,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":32.883000000000003,"5min":28.116,"15min":26.379999999999999},"min":{"1min":10.593,"5min":8.6489999999999991,"15min":7.2199999999999998},"max":{"1min":162.964,"5min":211.255,"15min":251.37700000000001},"last":8.7439999999999998},{"interface":"front","average":{"1min":38.536000000000001,"5min":33.917999999999999,"15min":28.524000000000001},"min":{"1min":9.3640000000000008,"5min":9.2349999999999994,"15min":7.1619999999999999},"max":{"1min":160.42599999999999,"5min":333.29700000000003,"15min":333.29700000000003},"last":17.245000000000001}]},{"osd":7,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":36.252000000000002,"5min":33.210000000000001,"15min":27.071999999999999},"min":{"1min":11.131,"5min":10.68,"15min":6.1769999999999996},"max":{"1min":116.877,"5min":258.81,"15min":264.52999999999997},"last":15.683999999999999},{"interface":"front","average":{"1min":26.748000000000001,"5min":22.239000000000001,"15min":25.803999999999998},"min":{"1min":7.4580000000000002,"5min":5.8410000000000002,"15min":5.8410000000000002},"max":{"1min":90.570999999999998,"5min":154.86799999999999,"15min":255.625},"last":11.167999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542408,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59852,"kb_used_data":6456,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311988,"statfs":{"total":96636764160,"available":96575475712,"internally_reserved":0,"allocated":6610944,"data_stored":3136155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":9.2010000000000005,"5min":13.369999999999999,"15min":14.285},"min":{"1min":4.9580000000000002,"5min":4.0499999999999998,"15min":3.5920000000000001},"max":{"1min":18.266999999999999,"5min":106.79300000000001,"15min":187.85499999999999},"last":18.699999999999999},{"interface":"front","average":{"1min":8.7729999999999997,"5min":12.766999999999999,"15min":13.725},"min":{"1min":5.0279999999999996,"5min":4.1020000000000003,"15min":4.1020000000000003},"max":{"1min":15.933,"5min":117.645,"15min":124.342},"last":7.9589999999999996}]},{"osd":1,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":13.706,"5min":18.475999999999999,"15min":19.167999999999999},"min":{"1min":8.1969999999999992,"5min":5.8979999999999997,"15min":5.7409999999999997},"max":{"1min":33.970999999999997,"5min":103.907,"15min":226.63399999999999},"last":19.806000000000001},{"interface":"front","average":{"1min":11.929,"5min":17.565999999999999,"15min":19.605},"min":{"1min":6.5439999999999996,"5min":6.351,"15min":5.859},"max":{"1min":42.719000000000001,"5min":135.41200000000001,"15min":187.24100000000001},"last":11.147}]},{"osd":2,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":17.675000000000001,"5min":18.248999999999999,"15min":20.965},"min":{"1min":7.2869999999999999,"5min":6.2549999999999999,"15min":6.2549999999999999},"max":{"1min":43.064999999999998,"5min":79.168000000000006,"15min":200.285},"last":13.004},{"interface":"front","average":{"1min":26.209,"5min":21.07,"15min":21.149999999999999},"min":{"1min":6.0679999999999996,"5min":5.5090000000000003,"15min":5.5090000000000003},"max":{"1min":211.315,"5min":211.315,"15min":241.15700000000001},"last":5.4720000000000004}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":25.942,"5min":21.164000000000001,"15min":22.437999999999999},"min":{"1min":10.516,"5min":8.1509999999999998,"15min":6.6849999999999996},"max":{"1min":205.821,"5min":205.821,"15min":262.43900000000002},"last":14.073},{"interface":"front","average":{"1min":26.780000000000001,"5min":22.042999999999999,"15min":23.443999999999999},"min":{"1min":7.5650000000000004,"5min":6.7460000000000004,"15min":5.9950000000000001},"max":{"1min":225.29900000000001,"5min":225.29900000000001,"15min":330.15300000000002},"last":10.228}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.153,"5min":21.227,"15min":22.390000000000001},"min":{"1min":9.9459999999999997,"5min":7.218,"15min":7.218},"max":{"1min":22.241,"5min":136.494,"15min":213.072},"last":15.015000000000001},{"interface":"front","average":{"1min":17.73,"5min":22.32,"15min":24.960999999999999},"min":{"1min":8.9540000000000006,"5min":8.9540000000000006,"15min":8.5530000000000008},"max":{"1min":28.087,"5min":104.538,"15min":239.76900000000001},"last":17.381}]},{"osd":5,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.374000000000001,"5min":18.962,"15min":22.965},"min":{"1min":10.206,"5min":8.2889999999999997,"15min":7.258},"max":{"1min":27.122,"5min":110.45099999999999,"15min":275.08100000000002},"last":12.374000000000001},{"interface":"front","average":{"1min":13.379,"5min":15.702999999999999,"15min":18.018000000000001},"min":{"1min":5.3410000000000002,"5min":5.3410000000000002,"15min":5.3410000000000002},"max":{"1min":40.390000000000001,"5min":94.302999999999997,"15min":129.935},"last":7.3300000000000001}]},{"osd":6,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":18.283000000000001,"5min":21.891999999999999,"15min":25.209},"min":{"1min":11.725,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":27.867999999999999,"5min":126.001,"15min":263.41399999999999},"last":16.436},{"interface":"front","average":{"1min":17.962,"5min":19.896000000000001,"15min":23.355},"min":{"1min":11.305999999999999,"5min":8.0229999999999997,"15min":8.0229999999999997},"max":{"1min":56.701000000000001,"5min":111.691,"15min":257.75900000000001},"last":20.581}]}]},{"osd":2,"up_from":14,"seq":60129542408,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82048,"kb_used_data":6448,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289792,"statfs":{"total":96636764160,"available":96552747008,"internally_reserved":0,"allocated":6602752,"data_stored":3130585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":65,"apply_latency_ms":65,"commit_latency_ns":65000000,"apply_latency_ns":65000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":19.824999999999999,"5min":15.673,"15min":19.306999999999999},"min":{"1min":4.6959999999999997,"5min":4.3170000000000002,"15min":4.3170000000000002},"max":{"1min":211.80500000000001,"5min":211.80500000000001,"15min":389.31400000000002},"last":269.779},{"interface":"front","average":{"1min":18.757999999999999,"5min":20.440999999999999,"15min":19.539000000000001},"min":{"1min":5.9269999999999996,"5min":5.3040000000000003,"15min":4.4589999999999996},"max":{"1min":213.93600000000001,"5min":295.87,"15min":295.87},"last":124.283}]},{"osd":1,"last update":"Mon Oct 30 20:45:48 2023","interfaces":[{"interface":"back","average":{"1min":14.757,"5min":16.274999999999999,"15min":22.295999999999999},"min":{"1min":6.4720000000000004,"5min":5.5060000000000002,"15min":5.2649999999999997},"max":{"1min":36.689,"5min":74.575000000000003,"15min":245.84999999999999},"last":9.8390000000000004},{"interface":"front","average":{"1min":17.843,"5min":15.342000000000001,"15min":20.946999999999999},"min":{"1min":7.9459999999999997,"5min":5.8399999999999999,"15min":5.3330000000000002},"max":{"1min":66.947999999999993,"5min":72.662999999999997,"15min":317.20699999999999},"last":13.996}]},{"osd":3,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.094000000000001,"5min":17.843,"15min":22.158000000000001},"min":{"1min":6.6040000000000001,"5min":6.1459999999999999,"15min":5.2619999999999996},"max":{"1min":58.442999999999998,"5min":253.02799999999999,"15min":311.03100000000001},"last":8.6120000000000001},{"interface":"front","average":{"1min":20.954000000000001,"5min":21.210999999999999,"15min":28.140999999999998},"min":{"1min":10.403,"5min":7.4969999999999999,"15min":6.2039999999999997},"max":{"1min":86.935000000000002,"5min":145.066,"15min":703},"last":10.464}]},{"osd":4,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":14.718,"5min":18.241,"15min":26.344999999999999},"min":{"1min":7.4029999999999996,"5min":6.4809999999999999,"15min":6.3460000000000001},"max":{"1min":43.838999999999999,"5min":123.06399999999999,"15min":709.85900000000004},"last":12.739000000000001},{"interface":"front","average":{"1min":16.712,"5min":19.233000000000001,"15min":22.349},"min":{"1min":6.2359999999999998,"5min":6.2359999999999998,"15min":6.2359999999999998},"max":{"1min":65.897000000000006,"5min":118.82599999999999,"15min":182.947},"last":28.228000000000002}]},{"osd":5,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":18.765000000000001,"5min":18.805,"15min":29.754000000000001},"min":{"1min":8.3539999999999992,"5min":7.2279999999999998,"15min":6.8460000000000001},"max":{"1min":63.963000000000001,"5min":116.352,"15min":874.22799999999995},"last":31.942},{"interface":"front","average":{"1min":17.289999999999999,"5min":18.369,"15min":24.251999999999999},"min":{"1min":6.9500000000000002,"5min":6.9500000000000002,"15min":6.9500000000000002},"max":{"1min":41.155000000000001,"5min":73.218999999999994,"15min":222.76400000000001},"last":26.239999999999998}]},{"osd":6,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":15.76,"5min":23.898,"15min":25.614999999999998},"min":{"1min":8.5389999999999997,"5min":8.5389999999999997,"15min":7.3049999999999997},"max":{"1min":37.296999999999997,"5min":196.654,"15min":238.84800000000001},"last":28.916},{"interface":"front","average":{"1min":18.132000000000001,"5min":23.280000000000001,"15min":29.577000000000002},"min":{"1min":8.3770000000000007,"5min":8.0670000000000002,"15min":7.577},"max":{"1min":37.128999999999998,"5min":161.30799999999999,"15min":266.48599999999999},"last":31.02}]},{"osd":7,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":19.760000000000002,"5min":20.834,"15min":26.890000000000001},"min":{"1min":9.7300000000000004,"5min":9.7300000000000004,"15min":9.7300000000000004},"max":{"1min":39.747,"5min":133.429,"15min":354.904},"last":30.41},{"interface":"front","average":{"1min":19.949999999999999,"5min":21.204000000000001,"15min":29.919},"min":{"1min":9.1600000000000001,"5min":7.3440000000000003,"15min":7.3440000000000003},"max":{"1min":75.688999999999993,"5min":153.803,"15min":865.25800000000004},"last":13.436}]}]},{"osd":4,"up_from":13,"seq":55834575114,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82072,"kb_used_data":6476,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289768,"statfs":{"total":96636764160,"available":96552722432,"internally_reserved":0,"allocated":6631424,"data_stored":3163348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":59,"apply_latency_ms":59,"commit_latency_ns":59000000,"apply_latency_ns":59000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":12.099,"5min":10.295,"15min":15.339},"min":{"1min":4.5940000000000003,"5min":3.8839999999999999,"15min":3.839},"max":{"1min":73.688000000000002,"5min":118.02500000000001,"15min":224.95099999999999},"last":8.1910000000000007},{"interface":"front","average":{"1min":14.191000000000001,"5min":11.253,"15min":15.319000000000001},"min":{"1min":4.407,"5min":4.407,"15min":4.0229999999999997},"max":{"1min":70.055999999999997,"5min":70.055999999999997,"15min":227.71600000000001},"last":7.0209999999999999}]},{"osd":1,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":19.260999999999999,"5min":15.923,"15min":20.707999999999998},"min":{"1min":6.3550000000000004,"5min":4.7960000000000003,"15min":4.7960000000000003},"max":{"1min":92.763000000000005,"5min":118.747,"15min":383.57499999999999},"last":7.4740000000000002},{"interface":"front","average":{"1min":20.491,"5min":19.440000000000001,"15min":22.885999999999999},"min":{"1min":8.3000000000000007,"5min":7.8970000000000002,"15min":7.5},"max":{"1min":59.844999999999999,"5min":110.129,"15min":228.34999999999999},"last":10.853999999999999}]},{"osd":2,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":17.518000000000001,"5min":13.864000000000001,"15min":21.329000000000001},"min":{"1min":8.2390000000000008,"5min":4.6420000000000003,"15min":4.6420000000000003},"max":{"1min":54.417999999999999,"5min":82.471999999999994,"15min":347.30799999999999},"last":15.742000000000001},{"interface":"front","average":{"1min":23.317,"5min":16.451000000000001,"15min":21.233000000000001},"min":{"1min":6.1369999999999996,"5min":6.1369999999999996,"15min":6.0090000000000003},"max":{"1min":121.45999999999999,"5min":121.45999999999999,"15min":183.03100000000001},"last":11.478}]},{"osd":3,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":16.106000000000002,"5min":17.736000000000001,"15min":24.536000000000001},"min":{"1min":6.4779999999999998,"5min":6.4779999999999998,"15min":6.2930000000000001},"max":{"1min":70.343999999999994,"5min":174.733,"15min":270.19400000000002},"last":12.321999999999999},{"interface":"front","average":{"1min":18.103999999999999,"5min":17.949999999999999,"15min":24.756},"min":{"1min":6.0220000000000002,"5min":6.0220000000000002,"15min":6.0220000000000002},"max":{"1min":63.222000000000001,"5min":177.87,"15min":339.79700000000003},"last":9.3510000000000009}]},{"osd":5,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":20.681000000000001,"5min":20.640999999999998,"15min":24.623999999999999},"min":{"1min":6.5839999999999996,"5min":6.5839999999999996,"15min":6.5839999999999996},"max":{"1min":158.541,"5min":158.541,"15min":253.80199999999999},"last":16.981999999999999},{"interface":"front","average":{"1min":17.707000000000001,"5min":16.873000000000001,"15min":22.225000000000001},"min":{"1min":6.1029999999999998,"5min":6.1029999999999998,"15min":6.1029999999999998},"max":{"1min":147.54400000000001,"5min":147.54400000000001,"15min":271.572},"last":13.77}]},{"osd":6,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":26.273,"5min":18.638999999999999,"15min":24.821999999999999},"min":{"1min":7.0999999999999996,"5min":6.4210000000000003,"15min":6.4210000000000003},"max":{"1min":245.447,"5min":245.447,"15min":254.44399999999999},"last":15.035},{"interface":"front","average":{"1min":18.928999999999998,"5min":19.975000000000001,"15min":28.715},"min":{"1min":10.25,"5min":9.0419999999999998,"15min":8.1639999999999997},"max":{"1min":72.599000000000004,"5min":74.463999999999999,"15min":291.62599999999998},"last":14.419}]},{"osd":7,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":14.513,"5min":16.260000000000002,"15min":22.474},"min":{"1min":8.1890000000000001,"5min":8.1890000000000001,"15min":6.2389999999999999},"max":{"1min":43.911999999999999,"5min":76.325000000000003,"15min":262.82100000000003},"last":17.632999999999999},{"interface":"front","average":{"1min":19.515000000000001,"5min":19.870999999999999,"15min":25.632000000000001},"min":{"1min":9.8350000000000009,"5min":9.8350000000000009,"15min":9.8350000000000009},"max":{"1min":61.091000000000001,"5min":61.091000000000001,"15min":244.37799999999999},"last":18.791}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T20:45:52.974 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T20:45:53.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.072+0000 7f741f27c640 10 monclient: get_monmap_and_config 2023-10-30T20:45:53.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.072+0000 7f741f27c640 10 monclient: build_initial_monmap 2023-10-30T20:45:53.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.072+0000 7f741f27c640 10 monclient: monmap: 2023-10-30T20:45:53.074 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:53.075 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:53.075 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:53.073849+0000 2023-10-30T20:45:53.075 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:53.073849+0000 2023-10-30T20:45:53.075 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:53.075 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:53.076 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:53.076 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:53.076 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:53.076 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:53.076 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:53.076 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:53.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.073+0000 7f741f27c640 1 Processor -- start 2023-10-30T20:45:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.073+0000 7f741f27c640 1 -- start start 2023-10-30T20:45:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.073+0000 7f741f27c640 10 monclient: init 2023-10-30T20:45:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741f27c640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741f27c640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-10-30T20:45:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741f27c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f741806c960 0x7f74180afe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:53.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741f27c640 10 monclient(hunting): picked mon.noname-b con 0x7f741806c960 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:53.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741f27c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f741806bd40 addr v1:172.21.15.69:6789/0 2023-10-30T20:45:53.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741f27c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7418145800 0x7f741814c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:53.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741f27c640 10 monclient(hunting): picked mon.noname-a con 0x7f7418145800 addr v2:172.21.15.69:3300/0 2023-10-30T20:45:53.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741f27c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741806cd40 con 0x7f741806bd40 2023-10-30T20:45:53.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741f27c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:53.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741f27c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7418067a10 con 0x7f7418145800 2023-10-30T20:45:53.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741f27c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:53.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741f27c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741806fdc0 con 0x7f741806c960 2023-10-30T20:45:53.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741f27c640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:53.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741f27c640 10 monclient(hunting): authenticate will time out at 2876.363068s 2023-10-30T20:45:53.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741d819640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7418145800 0x7f741814c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:53.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.074+0000 7f741d018640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f741806c960 0x7f74180afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:53.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.075+0000 7f741d018640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f741806c960 0x7f74180afe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41660/0 (socket says 172.21.15.69:41660) 2023-10-30T20:45:53.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.075+0000 7f741d018640 1 -- 172.21.15.69:0/2278766186 learned_addr learned my addr 172.21.15.69:0/2278766186 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:53.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.075+0000 7f741d018640 10 monclient(hunting): get_auth_request con 0x7f741806c960 auth_method 0 2023-10-30T20:45:53.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.075+0000 7f741d018640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:53.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.075+0000 7f741d018640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:53.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.075+0000 7f741d018640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:53.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.075+0000 7f741d819640 10 monclient(hunting): get_auth_request con 0x7f7418145800 auth_method 0 2023-10-30T20:45:53.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.075+0000 7f741d819640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:53.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.075+0000 7f741d819640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:53.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.075+0000 7f741d819640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:53.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.075+0000 7f741d018640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:53.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.075+0000 7f741d018640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:53.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.075+0000 7f741d018640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:53.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.076+0000 7f741d819640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:53.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.076+0000 7f741d819640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:53.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.076+0000 7f741d819640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:53.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.077+0000 7f741d018640 10 monclient(hunting): handle_auth_done global_id 18272 payload 274 2023-10-30T20:45:53.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.077+0000 7f741d018640 1 -- 172.21.15.69:0/2278766186 >> v2:172.21.15.69:3300/0 conn(0x7f7418145800 msgr2=0x7f741814c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:53.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.077+0000 7f741d018640 1 --2- 172.21.15.69:0/2278766186 >> v2:172.21.15.69:3300/0 conn(0x7f7418145800 0x7f741814c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:53.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.077+0000 7f741d018640 1 -- 172.21.15.69:0/2278766186 >> v1:172.21.15.69:6789/0 conn(0x7f741806bd40 legacy=0x7f74181450f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:53.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.077+0000 7f741d018640 10 monclient: _finish_hunting 0 2023-10-30T20:45:53.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.077+0000 7f741d018640 1 monclient: found mon.noname-b 2023-10-30T20:45:53.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.077+0000 7f741d018640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:53.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.077+0000 7f741d018640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:45:53.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.077+0000 7f741d018640 1 -- 172.21.15.69:0/2278766186 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741806ffc0 con 0x7f741806c960 2023-10-30T20:45:53.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.079+0000 7f741d018640 1 --2- 172.21.15.69:0/2278766186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f741806c960 0x7f74180afe00 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 crypto rx=0x7f7410009870 tx=0x7f741002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b2905b757caf83e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:53.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f740ffff640 1 -- 172.21.15.69:0/2278766186 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7410030840 con 0x7f741806c960 2023-10-30T20:45:53.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f740ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:53.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f740ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:53.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f740ffff640 10 monclient: dump: 2023-10-30T20:45:53.093 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:53.093 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:53.093 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:53.093 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:53.094 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:53.094 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:53.094 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:53.094 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:53.094 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:53.095 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:53.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f740ffff640 10 monclient: _finish_auth 0 2023-10-30T20:45:53.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f740ffff640 10 monclient: _check_auth_tickets 2023-10-30T20:45:53.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f740ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:53.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f740ffff640 1 -- 172.21.15.69:0/2278766186 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74100309b0 con 0x7f741806c960 2023-10-30T20:45:53.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f741f27c640 5 monclient: authenticate success, global_id 18272 2023-10-30T20:45:53.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f741f27c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:45:53.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f740ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:53.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f741f27c640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:53.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f740ffff640 1 -- 172.21.15.69:0/2278766186 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7410030d50 con 0x7f741806c960 2023-10-30T20:45:53.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f740ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:53.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f741f27c640 4 set_mon_vals no callback set 2023-10-30T20:45:53.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f740ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:45:53.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f740ffff640 10 monclient: dump: 2023-10-30T20:45:53.097 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:53.097 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:53.097 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:53.098 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:53.098 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:53.098 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:53.098 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:53.098 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:53.099 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f741f27c640 10 monclient: shutdown 2023-10-30T20:45:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f741f27c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f741f27c640 1 -- 172.21.15.69:0/2278766186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f741806c960 msgr2=0x7f74180afe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:53.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f741f27c640 1 --2- 172.21.15.69:0/2278766186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f741806c960 0x7f74180afe00 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 crypto rx=0x7f7410009870 tx=0x7f741002b980 comp rx=0 tx=0).stop 2023-10-30T20:45:53.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f741f27c640 1 -- 172.21.15.69:0/2278766186 shutdown_connections 2023-10-30T20:45:53.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f741f27c640 1 --2- 172.21.15.69:0/2278766186 >> v2:172.21.15.69:3300/0 conn(0x7f7418145800 0x7f741814c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:53.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f741f27c640 1 --2- 172.21.15.69:0/2278766186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f741806c960 0x7f74180afe00 unknown :-1 s=CLOSED pgs=3202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:53.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.083+0000 7f741f27c640 1 -- 172.21.15.69:0/2278766186 >> 172.21.15.69:0/2278766186 conn(0x7f74180b53a0 msgr2=0x7f74180b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:53.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.084+0000 7f741f27c640 1 -- 172.21.15.69:0/2278766186 shutdown_connections 2023-10-30T20:45:53.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.084+0000 7f741f27c640 1 -- 172.21.15.69:0/2278766186 wait complete. 2023-10-30T20:45:53.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.084+0000 7f741f27c640 10 monclient: build_initial_monmap 2023-10-30T20:45:53.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.084+0000 7f741f27c640 10 monclient: monmap: 2023-10-30T20:45:53.101 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:53.102 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:53.102 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:53.102 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:53.102 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:53.102 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:53.102 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:53.103 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:53.103 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:53.103 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:53.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.084+0000 7f741f27c640 1 librados: starting msgr at 2023-10-30T20:45:53.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.084+0000 7f741f27c640 1 librados: starting objecter 2023-10-30T20:45:53.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 1 Processor -- start 2023-10-30T20:45:53.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 1 -- start start 2023-10-30T20:45:53.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 1 librados: setting wanted keys 2023-10-30T20:45:53.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 1 librados: calling monclient init 2023-10-30T20:45:53.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 10 monclient: init 2023-10-30T20:45:53.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:53.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:45:53.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f741806bd40 0x7f74181cc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:53.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 10 monclient(hunting): picked mon.noname-b con 0x7f741806bd40 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:53.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f741806c960 0x7f7418140420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:53.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 10 monclient(hunting): picked mon.noname-c con 0x7f741806c960 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:53.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7418145800 0x7f7418140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:53.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 10 monclient(hunting): picked mon.noname-a con 0x7f7418145800 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:53.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:53.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7418145c70 con 0x7f7418145800 2023-10-30T20:45:53.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:53.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7418141070 con 0x7f741806bd40 2023-10-30T20:45:53.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:53.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7418141500 con 0x7f741806c960 2023-10-30T20:45:53.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:53.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.085+0000 7f741f27c640 10 monclient(hunting): authenticate will time out at 2876.374485s 2023-10-30T20:45:53.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d819640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7418145800 0x7f7418140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d018640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f741806bd40 0x7f74181cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d819640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7418145800 0x7f7418140960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50616/0 (socket says 172.21.15.69:50616) 2023-10-30T20:45:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d819640 1 -- 172.21.15.69:0/1864055475 learned_addr learned my addr 172.21.15.69:0/1864055475 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d819640 10 monclient(hunting): get_auth_request con 0x7f7418145800 auth_method 0 2023-10-30T20:45:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d819640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:53.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d819640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:53.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d819640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:53.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d018640 10 monclient(hunting): get_auth_request con 0x7f741806bd40 auth_method 0 2023-10-30T20:45:53.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d018640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:53.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d018640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:53.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d018640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:53.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d819640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:53.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d819640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:53.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.086+0000 7f741d819640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:53.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.087+0000 7f741d018640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:53.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.087+0000 7f741d018640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:53.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.087+0000 7f741d018640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:53.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.087+0000 7f741c817640 1 --2- 172.21.15.69:0/1864055475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f741806c960 0x7f7418140420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:53.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.088+0000 7f741c817640 10 monclient(hunting): get_auth_request con 0x7f741806c960 auth_method 0 2023-10-30T20:45:53.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.088+0000 7f741c817640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:53.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.088+0000 7f741c817640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:53.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.088+0000 7f741c817640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:53.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.089+0000 7f741c817640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:53.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.089+0000 7f741c817640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:53.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.089+0000 7f741c817640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:53.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.090+0000 7f741d018640 10 monclient(hunting): handle_auth_done global_id 21211 payload 824 2023-10-30T20:45:53.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.090+0000 7f741d018640 1 -- 172.21.15.69:0/1864055475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f741806c960 msgr2=0x7f7418140420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:53.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.090+0000 7f741d018640 1 --2- 172.21.15.69:0/1864055475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f741806c960 0x7f7418140420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:53.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.090+0000 7f741d018640 1 -- 172.21.15.69:0/1864055475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7418145800 msgr2=0x7f7418140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:53.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.090+0000 7f741d018640 1 --2- 172.21.15.69:0/1864055475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7418145800 0x7f7418140960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:53.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.090+0000 7f741d018640 10 monclient: _finish_hunting 0 2023-10-30T20:45:53.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.090+0000 7f741d018640 1 monclient: found mon.noname-b 2023-10-30T20:45:53.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.090+0000 7f741d018640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:53.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.090+0000 7f741d018640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:45:53.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.090+0000 7f741d018640 1 -- 172.21.15.69:0/1864055475 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74181da080 con 0x7f741806bd40 2023-10-30T20:45:53.115 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.093+0000 7f741d018640 1 --2- 172.21.15.69:0/1864055475 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f741806bd40 0x7f74181cc290 secure :-1 s=READY pgs=5044 cs=0 l=1 rev1=1 crypto rx=0x7f7410033000 tx=0x7f7410004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:53.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.093+0000 7f740dffb640 10 client.?.objecter ms_handle_connect 0x7f741806bd40 2023-10-30T20:45:53.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.093+0000 7f740dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:53.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f740dffb640 1 -- 172.21.15.69:0/1864055475 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7410039c90 con 0x7f741806bd40 2023-10-30T20:45:53.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f740dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:53.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f740dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:53.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f740dffb640 10 monclient: dump: 2023-10-30T20:45:53.117 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:53.117 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:53.117 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:53.117 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:53.118 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:53.118 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:53.118 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:53.118 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:53.118 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:53.118 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:53.119 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f740dffb640 10 monclient: _finish_auth 0 2023-10-30T20:45:53.119 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f740dffb640 10 monclient: _check_auth_tickets 2023-10-30T20:45:53.119 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f740dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:53.119 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f740dffb640 1 -- 172.21.15.69:0/1864055475 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7410039e20 con 0x7f741806bd40 2023-10-30T20:45:53.119 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f740dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:53.119 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f740dffb640 1 -- 172.21.15.69:0/1864055475 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7410043640 con 0x7f741806bd40 2023-10-30T20:45:53.120 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f741f27c640 5 monclient: authenticate success, global_id 21211 2023-10-30T20:45:53.120 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f741f27c640 10 monclient: _renew_subs 2023-10-30T20:45:53.120 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f741f27c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:53.120 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f741f27c640 1 -- 172.21.15.69:0/1864055475 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74181da2e0 con 0x7f741806bd40 2023-10-30T20:45:53.120 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f740f7fe640 4 set_mon_vals no callback set 2023-10-30T20:45:53.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f740dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:53.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f740dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:53.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f740dffb640 10 monclient: dump: 2023-10-30T20:45:53.121 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:53.121 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:53.122 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:53.122 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:53.122 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:53.122 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:53.122 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:53.123 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:53.123 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:53.123 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:53.123 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f741f27c640 10 client.21211.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:53.123 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f741f27c640 10 monclient: _renew_subs 2023-10-30T20:45:53.123 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f741f27c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:53.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f741f27c640 1 -- 172.21.15.69:0/1864055475 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74181da710 con 0x7f741806bd40 2023-10-30T20:45:53.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.097+0000 7f741f27c640 1 librados: init done 2023-10-30T20:45:53.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.098+0000 7f741f27c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:53.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.098+0000 7f741f27c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:53.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.098+0000 7f741f27c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:53.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.098+0000 7f741f27c640 1 -- 172.21.15.69:0/1864055475 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7418072280 con 0x7f741806bd40 2023-10-30T20:45:53.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.119+0000 7f740dffb640 1 -- 172.21.15.69:0/1864055475 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f74100437b0 con 0x7f741806bd40 2023-10-30T20:45:53.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.120+0000 7f740dffb640 1 --2- 172.21.15.69:0/1864055475 >> v2:172.21.15.69:6800/1876307353 conn(0x7f73fc06c840 0x7f73fc06ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:53.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.120+0000 7f741c817640 1 --2- 172.21.15.69:0/1864055475 >> v2:172.21.15.69:6800/1876307353 conn(0x7f73fc06c840 0x7f73fc06ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:53.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.120+0000 7f741c817640 10 monclient: get_auth_request con 0x7f73fc06c840 auth_method 0 2023-10-30T20:45:53.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.120+0000 7f740dffb640 10 client.21211.objecter ms_handle_connect 0x7f73fc06c840 2023-10-30T20:45:53.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.120+0000 7f741c817640 1 --2- 172.21.15.69:0/1864055475 >> v2:172.21.15.69:6800/1876307353 conn(0x7f73fc06c840 0x7f73fc06ed00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f741813fef0 tx=0x7f7408009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:53.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.342+0000 7f740dffb640 1 -- 172.21.15.69:0/1864055475 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(903..903 src has 451..903) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f7410030960 con 0x7f741806bd40 2023-10-30T20:45:53.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.342+0000 7f740dffb640 10 client.21211.objecter ms_dispatch 0x7f7418000b90 osd_map(903..903 src has 451..903) v4 2023-10-30T20:45:53.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.342+0000 7f740dffb640 3 client.21211.objecter handle_osd_map got epochs [903,903] > 0 2023-10-30T20:45:53.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.342+0000 7f740dffb640 3 client.21211.objecter handle_osd_map decoding full epoch 903 2023-10-30T20:45:53.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.342+0000 7f740dffb640 20 client.21211.objecter dump_active .. 0 homeless 2023-10-30T20:45:53.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.351+0000 7f740dffb640 1 -- 172.21.15.69:0/1864055475 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7410048070 con 0x7f741806bd40 2023-10-30T20:45:53.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.351+0000 7f740dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:53.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.351+0000 7f740dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:45:53.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.521+0000 7f741f27c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:45:53.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.521+0000 7f741f27c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:45:53.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.521+0000 7f741f27c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:53.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.521+0000 7f741f27c640 1 -- 172.21.15.69:0/1864055475 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f741806ffc0 con 0x7f741806bd40 2023-10-30T20:45:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.554+0000 7f740dffb640 1 -- 172.21.15.69:0/1864055475 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v903) v1 ==== 74+0+15629 (secure 0 0 0) 0x7f741007b330 con 0x7f741806bd40 2023-10-30T20:45:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.554+0000 7f740dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T20:45:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.554+0000 7f740dffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:45:53.556 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:45:53.556 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":903,"fsid":"32c1a723-7694-47d8-8b37-b1257dfe1796","created":"2023-10-30T20:23:06.559937+0000","modified":"2023-10-30T20:45:50.501283+0000","last_up_change":"2023-10-30T20:23:54.534733+0000","last_in_change":"2023-10-30T20:23:12.023084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":102,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T20:23:54.411443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T20:24:01.613740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"ceph_test_rados_api_asio","create_time":"2023-10-30T20:25:17.167018+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"af10bc15-318a-4349-af76-32a30c8acf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":896,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6801","nonce":2068724062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2068724062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":2068724062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":2068724062}]},"public_addr":"172.21.15.69:6801/2068724062","cluster_addr":"172.21.15.69:6802/2068724062","heartbeat_back_addr":"172.21.15.69:6804/2068724062","heartbeat_front_addr":"172.21.15.69:6803/2068724062","state":["exists","up"]},{"osd":1,"uuid":"d0d32a9a-e24a-4d0c-9b90-125485d70672","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":896,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6813","nonce":431025540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":431025540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":431025540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":431025540}]},"public_addr":"172.21.15.69:6813/431025540","cluster_addr":"172.21.15.69:6814/431025540","heartbeat_back_addr":"172.21.15.69:6816/431025540","heartbeat_front_addr":"172.21.15.69:6815/431025540","state":["exists","up"]},{"osd":2,"uuid":"508f2c3a-9e54-46aa-b6c0-52fad97fd341","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":896,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6805","nonce":2300065874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2300065874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2300065874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2300065874}]},"public_addr":"172.21.15.69:6805/2300065874","cluster_addr":"172.21.15.69:6806/2300065874","heartbeat_back_addr":"172.21.15.69:6808/2300065874","heartbeat_front_addr":"172.21.15.69:6807/2300065874","state":["exists","up"]},{"osd":3,"uuid":"003a2613-1050-4a96-b927-0e5983687319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":896,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6809","nonce":1824616966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1824616966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1824616966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1824616966}]},"public_addr":"172.21.15.69:6809/1824616966","cluster_addr":"172.21.15.69:6810/1824616966","heartbeat_back_addr":"172.21.15.69:6812/1824616966","heartbeat_front_addr":"172.21.15.69:6811/1824616966","state":["exists","up"]},{"osd":4,"uuid":"678fc58e-ed6a-4fd0-9825-93297e3af6f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":4151811482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6801","nonce":4151811482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6803","nonce":4151811482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":4151811482}]},"public_addr":"172.21.15.183:6800/4151811482","cluster_addr":"172.21.15.183:6801/4151811482","heartbeat_back_addr":"172.21.15.183:6803/4151811482","heartbeat_front_addr":"172.21.15.183:6802/4151811482","state":["exists","up"]},{"osd":5,"uuid":"f945159a-a53c-48ba-97df-9374f9e56ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":896,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2276632263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6813","nonce":2276632263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6815","nonce":2276632263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2276632263}]},"public_addr":"172.21.15.183:6812/2276632263","cluster_addr":"172.21.15.183:6813/2276632263","heartbeat_back_addr":"172.21.15.183:6815/2276632263","heartbeat_front_addr":"172.21.15.183:6814/2276632263","state":["exists","up"]},{"osd":6,"uuid":"8184f59d-15e2-4bdc-90e3-fe1e231ca125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":896,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1046221745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":1046221745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6807","nonce":1046221745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1046221745}]},"public_addr":"172.21.15.183:6804/1046221745","cluster_addr":"172.21.15.183:6805/1046221745","heartbeat_back_addr":"172.21.15.183:6807/1046221745","heartbeat_front_addr":"172.21.15.183:6806/1046221745","state":["exists","up"]},{"osd":7,"uuid":"1ebbf31b-eb42-4cd2-9044-a0bb0fc4f35d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":896,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1361328034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1361328034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":1361328034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1361328034}]},"public_addr":"172.21.15.183:6808/1361328034","cluster_addr":"172.21.15.183:6809/1361328034","heartbeat_back_addr":"172.21.15.183:6811/1361328034","heartbeat_front_addr":"172.21.15.183:6810/1361328034","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.509883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.602550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:50.691692+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.518736+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:48.903386+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:51.213129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:50.677675+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T20:23:50.250250+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"10.1b","mappings":[{"from":2,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":20,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":57,"snaps":[{"begin":6,"length":1},{"begin":11,"length":2}]},{"pool":84,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":96,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":100,"snaps":[{"begin":2,"length":1}]},{"pool":108,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":113,"snaps":[{"begin":4,"length":2}]},{"pool":119,"snaps":[{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":175,"snaps":[{"begin":12,"length":1}]},{"pool":186,"snaps":[{"begin":14,"length":1}]},{"pool":190,"snaps":[{"begin":17,"length":1}]},{"pool":193,"snaps":[{"begin":19,"length":2}]},{"pool":197,"snaps":[{"begin":24,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":4,"length":1}]},{"pool":244,"snaps":[{"begin":6,"length":1}]},{"pool":248,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T20:45:53.558 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 10 librados: watch_flush enter 2023-10-30T20:45:53.558 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 10 librados: watch_flush exit 2023-10-30T20:45:53.558 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 20 client.21211.objecter shutdown clearing up homeless session... 2023-10-30T20:45:53.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 10 client.21211.objecter successfully canceled tick 2023-10-30T20:45:53.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 1 -- 172.21.15.69:0/1864055475 >> v2:172.21.15.69:6800/1876307353 conn(0x7f73fc06c840 msgr2=0x7f73fc06ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:53.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 1 --2- 172.21.15.69:0/1864055475 >> v2:172.21.15.69:6800/1876307353 conn(0x7f73fc06c840 0x7f73fc06ed00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f741813fef0 tx=0x7f7408009040 comp rx=0 tx=0).stop 2023-10-30T20:45:53.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 10 monclient: shutdown 2023-10-30T20:45:53.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:53.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 1 -- 172.21.15.69:0/1864055475 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f741806bd40 msgr2=0x7f74181cc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:53.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 1 --2- 172.21.15.69:0/1864055475 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f741806bd40 0x7f74181cc290 secure :-1 s=READY pgs=5044 cs=0 l=1 rev1=1 crypto rx=0x7f7410033000 tx=0x7f7410004730 comp rx=0 tx=0).stop 2023-10-30T20:45:53.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 1 -- 172.21.15.69:0/1864055475 shutdown_connections 2023-10-30T20:45:53.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 1 --2- 172.21.15.69:0/1864055475 >> v2:172.21.15.69:6800/1876307353 conn(0x7f73fc06c840 0x7f73fc06ed00 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:53.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 1 --2- 172.21.15.69:0/1864055475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7418145800 0x7f7418140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 1 --2- 172.21.15.69:0/1864055475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f741806c960 0x7f7418140420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 1 --2- 172.21.15.69:0/1864055475 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f741806bd40 0x7f74181cc290 unknown :-1 s=CLOSED pgs=5044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 1 -- 172.21.15.69:0/1864055475 >> 172.21.15.69:0/1864055475 conn(0x7f74180b53a0 msgr2=0x7f74180737b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 1 -- 172.21.15.69:0/1864055475 shutdown_connections 2023-10-30T20:45:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 1 -- 172.21.15.69:0/1864055475 wait complete. 2023-10-30T20:45:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:53.556+0000 7f741f27c640 1 librados: shutdown 2023-10-30T20:45:54.570 INFO:tasks.ceph:Scrubbing osd.0 2023-10-30T20:45:54.571 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-30T20:45:54.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.674+0000 7fc8d524f640 10 monclient: get_monmap_and_config 2023-10-30T20:45:54.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.674+0000 7fc8d524f640 10 monclient: build_initial_monmap 2023-10-30T20:45:54.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.674+0000 7fc8d524f640 10 monclient: monmap: 2023-10-30T20:45:54.676 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:54.676 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:54.675127+0000 2023-10-30T20:45:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:54.675127+0000 2023-10-30T20:45:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:54.678 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:54.678 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:54.678 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:54.678 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:54.678 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:54.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.674+0000 7fc8d524f640 1 Processor -- start 2023-10-30T20:45:54.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.674+0000 7fc8d524f640 1 -- start start 2023-10-30T20:45:54.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.674+0000 7fc8d524f640 10 monclient: init 2023-10-30T20:45:54.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.675+0000 7fc8d524f640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:54.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.675+0000 7fc8d524f640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-10-30T20:45:54.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.675+0000 7fc8d524f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8d00b4420 addr v1:172.21.15.183:6789/0 2023-10-30T20:45:54.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.675+0000 7fc8d524f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8d00b12d0 0x7fc8d013f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:54.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.675+0000 7fc8d524f640 10 monclient(hunting): picked mon.noname-b con 0x7fc8d00b12d0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:54.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.675+0000 7fc8d524f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8d0141910 addr v1:172.21.15.69:6789/0 2023-10-30T20:45:54.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.675+0000 7fc8d524f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d00a9120 con 0x7fc8d0141910 2023-10-30T20:45:54.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.675+0000 7fc8d524f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d00aa460 con 0x7fc8d00b4420 2023-10-30T20:45:54.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.675+0000 7fc8d524f640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:54.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.675+0000 7fc8d524f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d0067a10 con 0x7fc8d00b12d0 2023-10-30T20:45:54.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.675+0000 7fc8d524f640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:54.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.675+0000 7fc8d524f640 10 monclient(hunting): authenticate will time out at 2877.964163s 2023-10-30T20:45:54.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.676+0000 7fc8cf59e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc8d0141910 0x7fc8d0141da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:38928/0 (socket says 172.21.15.69:38928) 2023-10-30T20:45:54.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.676+0000 7fc8ced9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fc8d00b4420 0x7fc8d013ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.69:55278/0 (socket says 172.21.15.69:55278) 2023-10-30T20:45:54.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.676+0000 7fc8cf59e640 1 -- 172.21.15.69:0/1617014510 learned_addr learned my addr 172.21.15.69:0/1617014510 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:45:54.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.676+0000 7fc8ce59c640 1 --2- 172.21.15.69:0/1617014510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8d00b12d0 0x7fc8d013f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:54.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.676+0000 7fc8ce59c640 10 monclient(hunting): get_auth_request con 0x7fc8d00b12d0 auth_method 0 2023-10-30T20:45:54.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.676+0000 7fc8ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:54.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.676+0000 7fc8ce59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:54.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.676+0000 7fc8ce59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:54.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.677+0000 7fc8ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:54.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.677+0000 7fc8ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:54.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.677+0000 7fc8ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:54.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8cdd9b640 1 -- 172.21.15.69:0/1617014510 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725717259 0 0) 0x7fc8c8001110 con 0x7fc8d0141910 2023-10-30T20:45:54.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8cdd9b640 10 monclient(hunting): _negotiate 2023-10-30T20:45:54.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8cdd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:54.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8cdd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:54.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8cdd9b640 10 monclient(hunting): my global_id is 16239 2023-10-30T20:45:54.687 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8cdd9b640 1 -- 172.21.15.69:0/1617014510 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b8003520 con 0x7fc8d0141910 2023-10-30T20:45:54.687 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8cdd9b640 1 -- 172.21.15.69:0/1617014510 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102266470 0 0) 0x7fc8c0001490 con 0x7fc8d00b4420 2023-10-30T20:45:54.687 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8cdd9b640 10 monclient(hunting): _negotiate 2023-10-30T20:45:54.687 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8cdd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:54.687 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8cdd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8cdd9b640 10 monclient(hunting): my global_id is 21214 2023-10-30T20:45:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8cdd9b640 1 -- 172.21.15.69:0/1617014510 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b8003840 con 0x7fc8d00b4420 2023-10-30T20:45:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8ce59c640 10 monclient(hunting): handle_auth_done global_id 18278 payload 274 2023-10-30T20:45:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8ce59c640 1 -- 172.21.15.69:0/1617014510 >> v1:172.21.15.183:6789/0 conn(0x7fc8d00b4420 legacy=0x7fc8d013ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:54.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8ce59c640 1 -- 172.21.15.69:0/1617014510 >> v1:172.21.15.69:6789/0 conn(0x7fc8d0141910 legacy=0x7fc8d0141da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:54.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8ce59c640 10 monclient: _finish_hunting 0 2023-10-30T20:45:54.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8ce59c640 1 monclient: found mon.noname-b 2023-10-30T20:45:54.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8ce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:54.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8ce59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:45:54.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.678+0000 7fc8ce59c640 1 -- 172.21.15.69:0/1617014510 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d00aa740 con 0x7fc8d00b12d0 2023-10-30T20:45:54.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.680+0000 7fc8ce59c640 1 --2- 172.21.15.69:0/1617014510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8d00b12d0 0x7fc8d013f160 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 crypto rx=0x7fc8c40097d0 tx=0x7fc8c402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f911b8a2909e5817 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:54.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8cdd9b640 1 -- 172.21.15.69:0/1617014510 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4031800 con 0x7fc8d00b12d0 2023-10-30T20:45:54.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:54.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8cdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:54.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8cdd9b640 10 monclient: dump: 2023-10-30T20:45:54.694 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:54.695 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:54.695 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.695 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.695 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:54.695 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:54.695 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:54.696 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:54.696 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:54.696 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:54.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8cdd9b640 10 monclient: _finish_auth 0 2023-10-30T20:45:54.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8cdd9b640 10 monclient: _check_auth_tickets 2023-10-30T20:45:54.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:54.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8cdd9b640 1 -- 172.21.15.69:0/1617014510 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c4031970 con 0x7fc8d00b12d0 2023-10-30T20:45:54.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8d524f640 5 monclient: authenticate success, global_id 18278 2023-10-30T20:45:54.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8d524f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:45:54.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8cdd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:54.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8cdd9b640 1 -- 172.21.15.69:0/1617014510 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4031d10 con 0x7fc8d00b12d0 2023-10-30T20:45:54.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:54.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8cdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:45:54.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8cdd9b640 10 monclient: dump: 2023-10-30T20:45:54.698 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:54.699 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:54.699 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.699 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.699 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:54.699 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:54.699 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:54.700 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:54.700 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:54.700 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:54.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8d524f640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:54.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8d524f640 4 set_mon_vals no callback set 2023-10-30T20:45:54.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8d524f640 10 monclient: shutdown 2023-10-30T20:45:54.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8d524f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:54.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8d524f640 1 -- 172.21.15.69:0/1617014510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8d00b12d0 msgr2=0x7fc8d013f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:54.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8d524f640 1 --2- 172.21.15.69:0/1617014510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8d00b12d0 0x7fc8d013f160 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 crypto rx=0x7fc8c40097d0 tx=0x7fc8c402b8a0 comp rx=0 tx=0).stop 2023-10-30T20:45:54.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8d524f640 1 -- 172.21.15.69:0/1617014510 shutdown_connections 2023-10-30T20:45:54.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8d524f640 1 --2- 172.21.15.69:0/1617014510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8d00b12d0 0x7fc8d013f160 unknown :-1 s=CLOSED pgs=3203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:54.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8d524f640 1 -- 172.21.15.69:0/1617014510 >> 172.21.15.69:0/1617014510 conn(0x7fc8d0065ea0 msgr2=0x7fc8d00662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:54.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8d524f640 1 -- 172.21.15.69:0/1617014510 shutdown_connections 2023-10-30T20:45:54.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.684+0000 7fc8d524f640 1 -- 172.21.15.69:0/1617014510 wait complete. 2023-10-30T20:45:54.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.685+0000 7fc8d524f640 10 monclient: build_initial_monmap 2023-10-30T20:45:54.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.685+0000 7fc8d524f640 10 monclient: monmap: 2023-10-30T20:45:54.703 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:54.703 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:54.703 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:54.703 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:54.703 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:54.704 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:54.704 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:54.704 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:54.704 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:54.704 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:54.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.685+0000 7fc8d524f640 1 librados: starting msgr at 2023-10-30T20:45:54.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.685+0000 7fc8d524f640 1 librados: starting objecter 2023-10-30T20:45:54.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 1 Processor -- start 2023-10-30T20:45:54.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 1 -- start start 2023-10-30T20:45:54.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 1 librados: setting wanted keys 2023-10-30T20:45:54.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 1 librados: calling monclient init 2023-10-30T20:45:54.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 10 monclient: init 2023-10-30T20:45:54.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:54.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T20:45:54.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8d00b12d0 0x7fc8d00b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:54.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 10 monclient(hunting): picked mon.noname-a con 0x7fc8d00b12d0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:54.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8d00b4420 0x7fc8d012fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:54.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 10 monclient(hunting): picked mon.noname-c con 0x7fc8d00b4420 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:54.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8d0141910 0x7fc8d0130060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:54.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 10 monclient(hunting): picked mon.noname-b con 0x7fc8d0141910 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:54.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:54.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00a8a90 con 0x7fc8d00b12d0 2023-10-30T20:45:54.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:54.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d0135e00 con 0x7fc8d0141910 2023-10-30T20:45:54.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:54.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d0136000 con 0x7fc8d00b4420 2023-10-30T20:45:54.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:54.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.686+0000 7fc8d524f640 10 monclient(hunting): authenticate will time out at 2877.975694s 2023-10-30T20:45:54.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8ce59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8d00b4420 0x7fc8d012fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:54.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8ced9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8d00b12d0 0x7fc8d00b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:54.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8cf59e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8d0141910 0x7fc8d0130060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:54.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8ce59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8d00b4420 0x7fc8d012fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41684/0 (socket says 172.21.15.69:41684) 2023-10-30T20:45:54.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8ce59c640 1 -- 172.21.15.69:0/620540838 learned_addr learned my addr 172.21.15.69:0/620540838 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:54.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8ce59c640 10 monclient(hunting): get_auth_request con 0x7fc8d00b4420 auth_method 0 2023-10-30T20:45:54.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:54.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8ce59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:54.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8ce59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:54.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8ced9d640 10 monclient(hunting): get_auth_request con 0x7fc8d00b12d0 auth_method 0 2023-10-30T20:45:54.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:54.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8ced9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:54.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8ced9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:54.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8cf59e640 10 monclient(hunting): get_auth_request con 0x7fc8d0141910 auth_method 0 2023-10-30T20:45:54.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8cf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:54.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8cf59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:54.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.687+0000 7fc8cf59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:54.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.688+0000 7fc8ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:54.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.688+0000 7fc8ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:54.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.688+0000 7fc8ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:54.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.688+0000 7fc8cf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:54.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.688+0000 7fc8cf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:54.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.688+0000 7fc8cf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:54.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.688+0000 7fc8ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:54.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.688+0000 7fc8ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:54.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.688+0000 7fc8ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:54.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.690+0000 7fc8ce59c640 10 monclient(hunting): handle_auth_done global_id 18281 payload 824 2023-10-30T20:45:54.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.690+0000 7fc8ce59c640 1 -- 172.21.15.69:0/620540838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8d0141910 msgr2=0x7fc8d0130060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:54.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.690+0000 7fc8ce59c640 1 --2- 172.21.15.69:0/620540838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8d0141910 0x7fc8d0130060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:54.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.691+0000 7fc8ce59c640 1 -- 172.21.15.69:0/620540838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8d00b12d0 msgr2=0x7fc8d00b7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:54.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.691+0000 7fc8ce59c640 1 --2- 172.21.15.69:0/620540838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8d00b12d0 0x7fc8d00b7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:54.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.691+0000 7fc8ce59c640 10 monclient: _finish_hunting 0 2023-10-30T20:45:54.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.691+0000 7fc8ce59c640 1 monclient: found mon.noname-c 2023-10-30T20:45:54.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.691+0000 7fc8ce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:54.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.691+0000 7fc8ce59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:45:54.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.691+0000 7fc8ce59c640 1 -- 172.21.15.69:0/620540838 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d01362e0 con 0x7fc8d00b4420 2023-10-30T20:45:54.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.693+0000 7fc8ce59c640 1 --2- 172.21.15.69:0/620540838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8d00b4420 0x7fc8d012fb20 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4034040 tx=0x7fc8c40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:54.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.693+0000 7fc8b77fe640 10 client.?.objecter ms_handle_connect 0x7fc8d00b4420 2023-10-30T20:45:54.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.693+0000 7fc8b77fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:54.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8b77fe640 1 -- 172.21.15.69:0/620540838 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c403ac60 con 0x7fc8d00b4420 2023-10-30T20:45:54.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:54.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:45:54.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8b77fe640 10 monclient: dump: 2023-10-30T20:45:54.718 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:54.718 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:54.718 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.718 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.718 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:54.719 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:54.719 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:54.719 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:54.719 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:54.720 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:54.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8b77fe640 10 monclient: _finish_auth 0 2023-10-30T20:45:54.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8b77fe640 10 monclient: _check_auth_tickets 2023-10-30T20:45:54.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:54.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8b77fe640 1 -- 172.21.15.69:0/620540838 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c403e030 con 0x7fc8d00b4420 2023-10-30T20:45:54.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8b77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:54.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8b77fe640 1 -- 172.21.15.69:0/620540838 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4044630 con 0x7fc8d00b4420 2023-10-30T20:45:54.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8d524f640 5 monclient: authenticate success, global_id 18281 2023-10-30T20:45:54.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8d524f640 10 monclient: _renew_subs 2023-10-30T20:45:54.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8d524f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:45:54.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8d524f640 1 -- 172.21.15.69:0/620540838 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d00b6b80 con 0x7fc8d00b4420 2023-10-30T20:45:54.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:54.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:45:54.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8b77fe640 10 monclient: dump: 2023-10-30T20:45:54.722 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:54.723 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:54.723 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.723 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.723 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:54.723 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:54.724 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:54.724 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:54.724 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:54.724 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:54.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8cd59a640 4 set_mon_vals no callback set 2023-10-30T20:45:54.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8d524f640 10 client.18281.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:54.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8d524f640 10 monclient: _renew_subs 2023-10-30T20:45:54.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8d524f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:45:54.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8d524f640 1 -- 172.21.15.69:0/620540838 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d0134e30 con 0x7fc8d00b4420 2023-10-30T20:45:54.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.696+0000 7fc8d524f640 1 librados: init done 2023-10-30T20:45:54.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.697+0000 7fc8d524f640 10 client.18281.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:54.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.697+0000 7fc8d524f640 15 client.18281.objecter _session_command_op_assign -1 1 2023-10-30T20:45:54.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.697+0000 7fc8d524f640 20 client.18281.objecter _get_session osd=-1 returning homeless 2023-10-30T20:45:54.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.697+0000 7fc8d524f640 10 client.18281.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:54.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.697+0000 7fc8d524f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:45:54.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.697+0000 7fc8d524f640 1 -- 172.21.15.69:0/620540838 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc8d0135090 con 0x7fc8d00b4420 2023-10-30T20:45:54.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.716+0000 7fc8b77fe640 1 -- 172.21.15.69:0/620540838 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fc8c404d450 con 0x7fc8d00b4420 2023-10-30T20:45:54.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.716+0000 7fc8b77fe640 1 --2- 172.21.15.69:0/620540838 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc8bc06c810 0x7fc8bc06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:54.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.717+0000 7fc8ced9d640 1 --2- 172.21.15.69:0/620540838 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc8bc06c810 0x7fc8bc06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:54.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.717+0000 7fc8ced9d640 10 monclient: get_auth_request con 0x7fc8bc06c810 auth_method 0 2023-10-30T20:45:54.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.717+0000 7fc8ced9d640 1 --2- 172.21.15.69:0/620540838 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc8bc06c810 0x7fc8bc06ecd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0001700 tx=0x7fc8c0007f10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:54.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.717+0000 7fc8b77fe640 10 client.18281.objecter ms_handle_connect 0x7fc8bc06c810 2023-10-30T20:45:54.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 1 -- 172.21.15.69:0/620540838 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(903..903 src has 451..903) v4 ==== 5186+0+0 (secure 0 0 0) 0x7fc8c407a010 con 0x7fc8d00b4420 2023-10-30T20:45:54.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 10 client.18281.objecter ms_dispatch 0x7fc8d0000b90 osd_map(903..903 src has 451..903) v4 2023-10-30T20:45:54.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 3 client.18281.objecter handle_osd_map got epochs [903,903] > 0 2023-10-30T20:45:54.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 3 client.18281.objecter handle_osd_map decoding full epoch 903 2023-10-30T20:45:54.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 10 client.18281.objecter checking command 1 2023-10-30T20:45:54.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 1 --2- 172.21.15.69:0/620540838 >> v2:172.21.15.69:6801/2068724062 conn(0x7fc8bc073c50 0x7fc8bc0760f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:54.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 20 client.18281.objecter _get_session s=0x7fc8c4044630 osd=0 3 2023-10-30T20:45:54.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 20 client.18281.objecter put_session s=0x7fc8c4044630 osd=0 3 2023-10-30T20:45:54.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 15 client.18281.objecter _session_command_op_remove -1 1 2023-10-30T20:45:54.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 20 client.18281.objecter _get_session s=0x7fc8c4044630 osd=0 3 2023-10-30T20:45:54.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 20 client.18281.objecter get_session s=0x7fc8c4044630 osd=0 3 2023-10-30T20:45:54.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 15 client.18281.objecter _session_command_op_assign 0 1 2023-10-30T20:45:54.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 20 client.18281.objecter put_session s=0x7fc8c4044630 osd=0 4 2023-10-30T20:45:54.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 10 client.18281.objecter _send_command 1 2023-10-30T20:45:54.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 1 -- 172.21.15.69:0/620540838 --> v2:172.21.15.69:6801/2068724062 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8c403ac60 con 0x7fc8bc073c50 2023-10-30T20:45:54.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.718+0000 7fc8b77fe640 20 client.18281.objecter dump_active .. 0 homeless 2023-10-30T20:45:54.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.719+0000 7fc8cf59e640 1 --2- 172.21.15.69:0/620540838 >> v2:172.21.15.69:6801/2068724062 conn(0x7fc8bc073c50 0x7fc8bc0760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:54.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.719+0000 7fc8cf59e640 10 monclient: get_auth_request con 0x7fc8bc073c50 auth_method 0 2023-10-30T20:45:54.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.720+0000 7fc8b77fe640 1 -- 172.21.15.69:0/620540838 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=903) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8d0135e00 con 0x7fc8d00b4420 2023-10-30T20:45:54.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.720+0000 7fc8b77fe640 10 monclient: handle_get_version_reply finishing 1 version 903 2023-10-30T20:45:54.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.722+0000 7fc8cf59e640 1 --2- 172.21.15.69:0/620540838 >> v2:172.21.15.69:6801/2068724062 conn(0x7fc8bc073c50 0x7fc8bc0760f0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:54.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.722+0000 7fc8b77fe640 10 client.18281.objecter ms_handle_connect 0x7fc8bc073c50 2023-10-30T20:45:54.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.745+0000 7fc8b77fe640 1 -- 172.21.15.69:0/620540838 <== osd.0 v2:172.21.15.69:6801/2068724062 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fc8c403ac60 con 0x7fc8bc073c50 2023-10-30T20:45:54.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.745+0000 7fc8b77fe640 10 client.18281.objecter ms_dispatch 0x7fc8d0000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:45:54.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.745+0000 7fc8b77fe640 10 client.18281.objecter _finish_command 1 = system:0 2023-10-30T20:45:54.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.745+0000 7fc8b77fe640 20 client.18281.objecter put_session s=0x7fc8c4044630 osd=0 4 2023-10-30T20:45:54.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.745+0000 7fc8b77fe640 15 client.18281.objecter _session_command_op_remove 0 1 2023-10-30T20:45:54.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.758+0000 7fc8d524f640 10 client.18281.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T20:45:54.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.758+0000 7fc8d524f640 15 client.18281.objecter _session_command_op_assign -1 2 2023-10-30T20:45:54.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.758+0000 7fc8d524f640 20 client.18281.objecter _get_session s=0x7fc8c4044630 osd=0 3 2023-10-30T20:45:54.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.758+0000 7fc8d524f640 20 client.18281.objecter put_session s=0x7fc8c4044630 osd=0 3 2023-10-30T20:45:54.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.758+0000 7fc8d524f640 20 client.18281.objecter _get_session s=0x7fc8c4044630 osd=0 3 2023-10-30T20:45:54.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.758+0000 7fc8d524f640 15 client.18281.objecter _session_command_op_remove -1 2 2023-10-30T20:45:54.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.758+0000 7fc8d524f640 20 client.18281.objecter get_session s=0x7fc8c4044630 osd=0 3 2023-10-30T20:45:54.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.758+0000 7fc8d524f640 15 client.18281.objecter _session_command_op_assign 0 2 2023-10-30T20:45:54.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.758+0000 7fc8d524f640 20 client.18281.objecter put_session s=0x7fc8c4044630 osd=0 4 2023-10-30T20:45:54.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.758+0000 7fc8d524f640 10 client.18281.objecter _send_command 2 2023-10-30T20:45:54.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.758+0000 7fc8d524f640 1 -- 172.21.15.69:0/620540838 --> v2:172.21.15.69:6801/2068724062 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc8d00a87e0 con 0x7fc8bc073c50 2023-10-30T20:45:54.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.771+0000 7fc8b77fe640 1 -- 172.21.15.69:0/620540838 <== osd.0 v2:172.21.15.69:6801/2068724062 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc8c8006980 con 0x7fc8bc073c50 2023-10-30T20:45:54.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.771+0000 7fc8b77fe640 10 client.18281.objecter ms_dispatch 0x7fc8d0000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:45:54.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.771+0000 7fc8b77fe640 10 client.18281.objecter _finish_command 2 = system:0 2023-10-30T20:45:54.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.771+0000 7fc8b77fe640 20 client.18281.objecter put_session s=0x7fc8c4044630 osd=0 4 2023-10-30T20:45:54.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.771+0000 7fc8b77fe640 15 client.18281.objecter _session_command_op_remove 0 2 2023-10-30T20:45:54.774 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-10-30T20:45:54.774 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T20:45:54.774 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-10-30T20:45:54.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 10 librados: watch_flush enter 2023-10-30T20:45:54.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 10 librados: watch_flush exit 2023-10-30T20:45:54.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 10 client.18281.objecter close_session for osd.0 2023-10-30T20:45:54.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 1 -- 172.21.15.69:0/620540838 >> v2:172.21.15.69:6801/2068724062 conn(0x7fc8bc073c50 msgr2=0x7fc8bc0760f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:54.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 1 --2- 172.21.15.69:0/620540838 >> v2:172.21.15.69:6801/2068724062 conn(0x7fc8bc073c50 0x7fc8bc0760f0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:54.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 20 client.18281.objecter put_session s=0x7fc8c4044630 osd=0 1 2023-10-30T20:45:54.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 20 client.18281.objecter shutdown clearing up homeless session... 2023-10-30T20:45:54.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 10 client.18281.objecter successfully canceled tick 2023-10-30T20:45:54.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 1 -- 172.21.15.69:0/620540838 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc8bc06c810 msgr2=0x7fc8bc06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:54.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 1 --2- 172.21.15.69:0/620540838 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc8bc06c810 0x7fc8bc06ecd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0001700 tx=0x7fc8c0007f10 comp rx=0 tx=0).stop 2023-10-30T20:45:54.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 10 monclient: shutdown 2023-10-30T20:45:54.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:54.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 1 -- 172.21.15.69:0/620540838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8d00b4420 msgr2=0x7fc8d012fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:54.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 1 --2- 172.21.15.69:0/620540838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8d00b4420 0x7fc8d012fb20 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4034040 tx=0x7fc8c40046c0 comp rx=0 tx=0).stop 2023-10-30T20:45:54.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8cf59e640 1 -- 172.21.15.69:0/620540838 reap_dead start 2023-10-30T20:45:54.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 1 -- 172.21.15.69:0/620540838 shutdown_connections 2023-10-30T20:45:54.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 1 -- 172.21.15.69:0/620540838 >> 172.21.15.69:0/620540838 conn(0x7fc8d0065ea0 msgr2=0x7fc8d0141d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:54.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 1 -- 172.21.15.69:0/620540838 shutdown_connections 2023-10-30T20:45:54.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 1 -- 172.21.15.69:0/620540838 wait complete. 2023-10-30T20:45:54.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.772+0000 7fc8d524f640 1 librados: shutdown 2023-10-30T20:45:54.784 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-30T20:45:54.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.882+0000 7fc62beb4640 10 monclient: get_monmap_and_config 2023-10-30T20:45:54.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.882+0000 7fc62beb4640 10 monclient: build_initial_monmap 2023-10-30T20:45:54.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.882+0000 7fc62beb4640 10 monclient: monmap: 2023-10-30T20:45:54.884 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:54.885 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:54.885 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:54.883864+0000 2023-10-30T20:45:54.885 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:54.883864+0000 2023-10-30T20:45:54.885 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:54.886 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:54.886 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:54.886 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:54.886 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:54.886 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:54.887 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:54.887 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:54.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.883+0000 7fc62beb4640 1 Processor -- start 2023-10-30T20:45:54.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.883+0000 7fc62beb4640 1 -- start start 2023-10-30T20:45:54.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.883+0000 7fc62beb4640 10 monclient: init 2023-10-30T20:45:54.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:54.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-10-30T20:45:54.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc624071b60 0x7fc62413e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 10 monclient(hunting): picked mon.noname-c con 0x7fc624071b60 addr v2:172.21.15.183:3300/0 2023-10-30T20:45:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc6240aa810 0x7fc62413ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 10 monclient(hunting): picked mon.noname-a con 0x7fc6240aa810 addr v2:172.21.15.69:3300/0 2023-10-30T20:45:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6240ac8a0 0x7fc6240acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 10 monclient(hunting): picked mon.noname-b con 0x7fc6240ac8a0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc624074330 con 0x7fc6240aa810 2023-10-30T20:45:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6240af8a0 con 0x7fc624071b60 2023-10-30T20:45:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6240ac1a0 con 0x7fc6240ac8a0 2023-10-30T20:45:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62beb4640 10 monclient(hunting): authenticate will time out at 2878.173064s 2023-10-30T20:45:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62a451640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6240ac8a0 0x7fc6240acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.884+0000 7fc62944f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc6240aa810 0x7fc62413ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc629c50640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc624071b60 0x7fc62413e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62944f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc6240aa810 0x7fc62413ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50630/0 (socket says 172.21.15.69:50630) 2023-10-30T20:45:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62a451640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6240ac8a0 0x7fc6240acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41694/0 (socket says 172.21.15.69:41694) 2023-10-30T20:45:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62944f640 1 -- 172.21.15.69:0/637971850 learned_addr learned my addr 172.21.15.69:0/637971850 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62944f640 10 monclient(hunting): get_auth_request con 0x7fc6240aa810 auth_method 0 2023-10-30T20:45:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62944f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62944f640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62944f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62a451640 10 monclient(hunting): get_auth_request con 0x7fc6240ac8a0 auth_method 0 2023-10-30T20:45:54.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62a451640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:54.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62a451640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:54.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62a451640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:54.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc629c50640 10 monclient(hunting): get_auth_request con 0x7fc624071b60 auth_method 0 2023-10-30T20:45:54.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc629c50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:54.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc629c50640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:54.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc629c50640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:54.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62a451640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:54.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62a451640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:54.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62a451640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:54.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62944f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:54.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62944f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:54.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.885+0000 7fc62944f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:54.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.886+0000 7fc629c50640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:54.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.886+0000 7fc629c50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:54.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.886+0000 7fc629c50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:54.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.887+0000 7fc62a451640 10 monclient(hunting): handle_auth_done global_id 18284 payload 274 2023-10-30T20:45:54.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.887+0000 7fc62a451640 1 -- 172.21.15.69:0/637971850 >> v2:172.21.15.183:3300/0 conn(0x7fc624071b60 msgr2=0x7fc62413e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:54.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.887+0000 7fc62a451640 1 --2- 172.21.15.69:0/637971850 >> v2:172.21.15.183:3300/0 conn(0x7fc624071b60 0x7fc62413e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:54.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.887+0000 7fc62a451640 1 -- 172.21.15.69:0/637971850 >> v2:172.21.15.69:3300/0 conn(0x7fc6240aa810 msgr2=0x7fc62413ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:54.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.887+0000 7fc62a451640 1 --2- 172.21.15.69:0/637971850 >> v2:172.21.15.69:3300/0 conn(0x7fc6240aa810 0x7fc62413ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:54.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.887+0000 7fc62a451640 10 monclient: _finish_hunting 0 2023-10-30T20:45:54.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.887+0000 7fc62a451640 1 monclient: found mon.noname-b 2023-10-30T20:45:54.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.887+0000 7fc62a451640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:54.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.887+0000 7fc62a451640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:45:54.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.887+0000 7fc62a451640 1 -- 172.21.15.69:0/637971850 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6240ac480 con 0x7fc6240ac8a0 2023-10-30T20:45:54.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.887+0000 7fc62944f640 1 --2- 172.21.15.69:0/637971850 >> v2:172.21.15.69:3300/0 conn(0x7fc6240aa810 0x7fc62413ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T20:45:54.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.889+0000 7fc62a451640 1 --2- 172.21.15.69:0/637971850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6240ac8a0 0x7fc6240acd50 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 crypto rx=0x7fc6240afb40 tx=0x7fc61c02bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8f0254ab883a81b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:54.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc628c4e640 1 -- 172.21.15.69:0/637971850 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61c02cb40 con 0x7fc6240ac8a0 2023-10-30T20:45:54.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc628c4e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:54.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc628c4e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:54.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc628c4e640 10 monclient: dump: 2023-10-30T20:45:54.906 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:54.906 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:54.906 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.906 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.907 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:54.907 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:54.907 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:54.907 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:54.907 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:54.908 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:54.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc628c4e640 10 monclient: _finish_auth 0 2023-10-30T20:45:54.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc628c4e640 10 monclient: _check_auth_tickets 2023-10-30T20:45:54.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc628c4e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:54.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc628c4e640 1 -- 172.21.15.69:0/637971850 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61c02ccb0 con 0x7fc6240ac8a0 2023-10-30T20:45:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc62beb4640 5 monclient: authenticate success, global_id 18284 2023-10-30T20:45:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc62beb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:45:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc628c4e640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc628c4e640 1 -- 172.21.15.69:0/637971850 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61c034660 con 0x7fc6240ac8a0 2023-10-30T20:45:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc628c4e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc628c4e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:45:54.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc628c4e640 10 monclient: dump: 2023-10-30T20:45:54.910 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:54.910 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:54.910 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.910 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.911 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:54.911 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:54.911 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:54.911 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:54.911 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:54.912 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:54.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc62beb4640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:54.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc62beb4640 4 set_mon_vals no callback set 2023-10-30T20:45:54.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc62beb4640 10 monclient: shutdown 2023-10-30T20:45:54.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc62beb4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:54.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc62beb4640 1 -- 172.21.15.69:0/637971850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6240ac8a0 msgr2=0x7fc6240acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:54.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc62beb4640 1 --2- 172.21.15.69:0/637971850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6240ac8a0 0x7fc6240acd50 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 crypto rx=0x7fc6240afb40 tx=0x7fc61c02bd90 comp rx=0 tx=0).stop 2023-10-30T20:45:54.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc62beb4640 1 -- 172.21.15.69:0/637971850 shutdown_connections 2023-10-30T20:45:54.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc62beb4640 1 --2- 172.21.15.69:0/637971850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6240ac8a0 0x7fc6240acd50 unknown :-1 s=CLOSED pgs=3205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:54.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc62beb4640 1 --2- 172.21.15.69:0/637971850 >> v2:172.21.15.69:3300/0 conn(0x7fc6240aa810 0x7fc62413ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:54.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc62beb4640 1 --2- 172.21.15.69:0/637971850 >> v2:172.21.15.183:3300/0 conn(0x7fc624071b60 0x7fc62413e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:54.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.893+0000 7fc62beb4640 1 -- 172.21.15.69:0/637971850 >> 172.21.15.69:0/637971850 conn(0x7fc6240b3580 msgr2=0x7fc6240b3990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:54.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.894+0000 7fc62beb4640 1 -- 172.21.15.69:0/637971850 shutdown_connections 2023-10-30T20:45:54.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.894+0000 7fc62beb4640 1 -- 172.21.15.69:0/637971850 wait complete. 2023-10-30T20:45:54.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.894+0000 7fc62beb4640 10 monclient: build_initial_monmap 2023-10-30T20:45:54.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.894+0000 7fc62beb4640 10 monclient: monmap: 2023-10-30T20:45:54.915 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:54.915 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:54.915 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:54.915 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:54.915 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:54.916 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:54.916 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:54.916 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:54.916 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:54.916 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:54.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.894+0000 7fc62beb4640 1 librados: starting msgr at 2023-10-30T20:45:54.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.894+0000 7fc62beb4640 1 librados: starting objecter 2023-10-30T20:45:54.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 1 Processor -- start 2023-10-30T20:45:54.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 1 -- start start 2023-10-30T20:45:54.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 1 librados: setting wanted keys 2023-10-30T20:45:54.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 1 librados: calling monclient init 2023-10-30T20:45:54.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 10 monclient: init 2023-10-30T20:45:54.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:54.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T20:45:54.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc624071b60 0x7fc624144f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:54.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 10 monclient(hunting): picked mon.noname-b con 0x7fc624071b60 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:54.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6240aa810 0x7fc6240b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:54.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 10 monclient(hunting): picked mon.noname-a con 0x7fc6240aa810 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:54.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6240ac8a0 0x7fc6241413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:54.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 10 monclient(hunting): picked mon.noname-c con 0x7fc6240ac8a0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:54.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:54.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6240aedb0 con 0x7fc6240aa810 2023-10-30T20:45:54.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:54.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc624143850 con 0x7fc624071b60 2023-10-30T20:45:54.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:54.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc624141a10 con 0x7fc6240ac8a0 2023-10-30T20:45:54.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:54.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.895+0000 7fc62beb4640 10 monclient(hunting): authenticate will time out at 2878.184498s 2023-10-30T20:45:54.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc62a451640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6240ac8a0 0x7fc6241413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:54.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc62944f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6240aa810 0x7fc6240b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:54.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc629c50640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc624071b60 0x7fc624144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:54.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc62a451640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6240ac8a0 0x7fc6241413f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41710/0 (socket says 172.21.15.69:41710) 2023-10-30T20:45:54.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc62a451640 1 -- 172.21.15.69:0/1582651382 learned_addr learned my addr 172.21.15.69:0/1582651382 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:54.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc62a451640 10 monclient(hunting): get_auth_request con 0x7fc6240ac8a0 auth_method 0 2023-10-30T20:45:54.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc62a451640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:54.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc62a451640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:54.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc62a451640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:54.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc62944f640 10 monclient(hunting): get_auth_request con 0x7fc6240aa810 auth_method 0 2023-10-30T20:45:54.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc62944f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:54.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc62944f640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:54.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc62944f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:54.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc629c50640 10 monclient(hunting): get_auth_request con 0x7fc624071b60 auth_method 0 2023-10-30T20:45:54.923 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc629c50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:54.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc629c50640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:54.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.896+0000 7fc629c50640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:54.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.897+0000 7fc62a451640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:54.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.897+0000 7fc62a451640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:54.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.897+0000 7fc62a451640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:54.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.897+0000 7fc62944f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:54.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.897+0000 7fc62944f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:54.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.897+0000 7fc62944f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:54.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.897+0000 7fc629c50640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:54.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.897+0000 7fc629c50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:54.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.897+0000 7fc629c50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:54.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.900+0000 7fc629c50640 10 monclient(hunting): handle_auth_done global_id 21223 payload 824 2023-10-30T20:45:54.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.900+0000 7fc629c50640 1 -- 172.21.15.69:0/1582651382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6240ac8a0 msgr2=0x7fc6241413f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:54.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.900+0000 7fc629c50640 1 --2- 172.21.15.69:0/1582651382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6240ac8a0 0x7fc6241413f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:54.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.900+0000 7fc629c50640 1 -- 172.21.15.69:0/1582651382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6240aa810 msgr2=0x7fc6240b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:54.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.900+0000 7fc629c50640 1 --2- 172.21.15.69:0/1582651382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6240aa810 0x7fc6240b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:54.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.900+0000 7fc629c50640 10 monclient: _finish_hunting 0 2023-10-30T20:45:54.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.900+0000 7fc629c50640 1 monclient: found mon.noname-b 2023-10-30T20:45:54.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.900+0000 7fc629c50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:54.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.900+0000 7fc629c50640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:45:54.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.900+0000 7fc629c50640 1 -- 172.21.15.69:0/1582651382 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc624143f50 con 0x7fc624071b60 2023-10-30T20:45:54.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.902+0000 7fc629c50640 1 --2- 172.21.15.69:0/1582651382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc624071b60 0x7fc624144f40 secure :-1 s=READY pgs=5046 cs=0 l=1 rev1=1 crypto rx=0x7fc62000e850 tx=0x7fc62000ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:54.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.902+0000 7fc61a7fc640 10 client.?.objecter ms_handle_connect 0x7fc624071b60 2023-10-30T20:45:54.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.902+0000 7fc61a7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:54.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc61a7fc640 1 -- 172.21.15.69:0/1582651382 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62000c940 con 0x7fc624071b60 2023-10-30T20:45:54.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc61a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:54.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc61a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:54.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc61a7fc640 10 monclient: dump: 2023-10-30T20:45:54.931 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:54.931 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:54.931 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.931 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.932 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:54.932 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:54.932 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:54.932 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:54.932 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:54.933 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:54.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc61a7fc640 10 monclient: _finish_auth 0 2023-10-30T20:45:54.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc61a7fc640 10 monclient: _check_auth_tickets 2023-10-30T20:45:54.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc61a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:54.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc61a7fc640 1 -- 172.21.15.69:0/1582651382 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62000d040 con 0x7fc624071b60 2023-10-30T20:45:54.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc62beb4640 5 monclient: authenticate success, global_id 21223 2023-10-30T20:45:54.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc62beb4640 10 monclient: _renew_subs 2023-10-30T20:45:54.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc62beb4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:54.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc62beb4640 1 -- 172.21.15.69:0/1582651382 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc624137db0 con 0x7fc624071b60 2023-10-30T20:45:54.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc61a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:54.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc61a7fc640 1 -- 172.21.15.69:0/1582651382 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62000ccd0 con 0x7fc624071b60 2023-10-30T20:45:54.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc61a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:54.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc61a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:54.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc61a7fc640 10 monclient: dump: 2023-10-30T20:45:54.935 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:54.935 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:54.936 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.936 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:54.936 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:54.936 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:54.936 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:54.937 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:54.937 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:54.937 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:54.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc61bfff640 4 set_mon_vals no callback set 2023-10-30T20:45:54.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc62beb4640 10 client.21223.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:54.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc62beb4640 10 monclient: _renew_subs 2023-10-30T20:45:54.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc62beb4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:54.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc62beb4640 1 -- 172.21.15.69:0/1582651382 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc624138320 con 0x7fc624071b60 2023-10-30T20:45:54.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.906+0000 7fc62beb4640 1 librados: init done 2023-10-30T20:45:54.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.907+0000 7fc62beb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:54.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.907+0000 7fc62beb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.907+0000 7fc62beb4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.907+0000 7fc62beb4640 1 -- 172.21.15.69:0/1582651382 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc624143c90 con 0x7fc624071b60 2023-10-30T20:45:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.924+0000 7fc61a7fc640 1 -- 172.21.15.69:0/1582651382 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fc620013470 con 0x7fc624071b60 2023-10-30T20:45:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.924+0000 7fc61a7fc640 1 --2- 172.21.15.69:0/1582651382 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc60806c7c0 0x7fc60806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.925+0000 7fc62944f640 1 --2- 172.21.15.69:0/1582651382 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc60806c7c0 0x7fc60806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.925+0000 7fc62944f640 10 monclient: get_auth_request con 0x7fc60806c7c0 auth_method 0 2023-10-30T20:45:54.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.925+0000 7fc61a7fc640 10 client.21223.objecter ms_handle_connect 0x7fc60806c7c0 2023-10-30T20:45:54.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.925+0000 7fc62944f640 1 --2- 172.21.15.69:0/1582651382 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc60806c7c0 0x7fc60806ec80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc614009630 tx=0x7fc614009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:54.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.927+0000 7fc61a7fc640 1 -- 172.21.15.69:0/1582651382 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(903..903 src has 451..903) v4 ==== 5186+0+0 (secure 0 0 0) 0x7fc620058510 con 0x7fc624071b60 2023-10-30T20:45:54.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.927+0000 7fc61a7fc640 10 client.21223.objecter ms_dispatch 0x7fc624000b90 osd_map(903..903 src has 451..903) v4 2023-10-30T20:45:54.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.927+0000 7fc61a7fc640 3 client.21223.objecter handle_osd_map got epochs [903,903] > 0 2023-10-30T20:45:54.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.927+0000 7fc61a7fc640 3 client.21223.objecter handle_osd_map decoding full epoch 903 2023-10-30T20:45:54.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:54.927+0000 7fc61a7fc640 20 client.21223.objecter dump_active .. 0 homeless 2023-10-30T20:45:55.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.160+0000 7fc61a7fc640 1 -- 172.21.15.69:0/1582651382 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc62000ce40 con 0x7fc624071b60 2023-10-30T20:45:55.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.160+0000 7fc61a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:55.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.160+0000 7fc61a7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:45:55.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.342+0000 7fc62beb4640 1 -- 172.21.15.69:0/1582651382 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6240ad290 con 0x7fc60806c7c0 2023-10-30T20:45:55.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.343+0000 7fc61a7fc640 1 -- 172.21.15.69:0/1582651382 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc614011a90 con 0x7fc60806c7c0 2023-10-30T20:45:55.344 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 0 to deep-scrub 2023-10-30T20:45:55.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 10 librados: watch_flush enter 2023-10-30T20:45:55.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 10 librados: watch_flush exit 2023-10-30T20:45:55.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 20 client.21223.objecter shutdown clearing up homeless session... 2023-10-30T20:45:55.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 10 client.21223.objecter successfully canceled tick 2023-10-30T20:45:55.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 1 -- 172.21.15.69:0/1582651382 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc60806c7c0 msgr2=0x7fc60806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:55.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 1 --2- 172.21.15.69:0/1582651382 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc60806c7c0 0x7fc60806ec80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc614009630 tx=0x7fc614009240 comp rx=0 tx=0).stop 2023-10-30T20:45:55.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 10 monclient: shutdown 2023-10-30T20:45:55.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:55.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 1 -- 172.21.15.69:0/1582651382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc624071b60 msgr2=0x7fc624144f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:55.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 1 --2- 172.21.15.69:0/1582651382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc624071b60 0x7fc624144f40 secure :-1 s=READY pgs=5046 cs=0 l=1 rev1=1 crypto rx=0x7fc62000e850 tx=0x7fc62000ed10 comp rx=0 tx=0).stop 2023-10-30T20:45:55.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 1 -- 172.21.15.69:0/1582651382 shutdown_connections 2023-10-30T20:45:55.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 1 --2- 172.21.15.69:0/1582651382 >> v2:172.21.15.69:6800/1876307353 conn(0x7fc60806c7c0 0x7fc60806ec80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 1 --2- 172.21.15.69:0/1582651382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6240ac8a0 0x7fc6241413f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 1 --2- 172.21.15.69:0/1582651382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6240aa810 0x7fc6240b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.345+0000 7fc62beb4640 1 --2- 172.21.15.69:0/1582651382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc624071b60 0x7fc624144f40 unknown :-1 s=CLOSED pgs=5046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.346+0000 7fc62beb4640 1 -- 172.21.15.69:0/1582651382 >> 172.21.15.69:0/1582651382 conn(0x7fc6240b3580 msgr2=0x7fc62413d720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:55.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.346+0000 7fc62beb4640 1 -- 172.21.15.69:0/1582651382 shutdown_connections 2023-10-30T20:45:55.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.346+0000 7fc62beb4640 1 -- 172.21.15.69:0/1582651382 wait complete. 2023-10-30T20:45:55.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.346+0000 7fc62beb4640 1 librados: shutdown 2023-10-30T20:45:55.359 INFO:tasks.ceph:Scrubbing osd.1 2023-10-30T20:45:55.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-30T20:45:55.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.466+0000 7f72c99b7640 10 monclient: get_monmap_and_config 2023-10-30T20:45:55.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.466+0000 7f72c99b7640 10 monclient: build_initial_monmap 2023-10-30T20:45:55.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.466+0000 7f72c99b7640 10 monclient: monmap: 2023-10-30T20:45:55.468 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:55.468 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:55.468 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:55.467444+0000 2023-10-30T20:45:55.469 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:55.467444+0000 2023-10-30T20:45:55.469 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:55.469 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:55.469 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:55.469 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:55.469 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:55.470 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:55.470 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:55.470 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:55.472 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 1 Processor -- start 2023-10-30T20:45:55.472 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 1 -- start start 2023-10-30T20:45:55.472 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 10 monclient: init 2023-10-30T20:45:55.472 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-10-30T20:45:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72c40a8850 addr v1:172.21.15.183:6789/0 2023-10-30T20:45:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f72c40b1b90 0x7f72c40a9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 10 monclient(hunting): picked mon.noname-a con 0x7f72c40b1b90 addr v2:172.21.15.69:3300/0 2023-10-30T20:45:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c413eec0 0x7f72c413f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:55.474 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 10 monclient(hunting): picked mon.noname-b con 0x7f72c413eec0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:55.474 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c40aa7b0 con 0x7f72c40a8850 2023-10-30T20:45:55.474 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:55.474 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72c4074040 con 0x7f72c40b1b90 2023-10-30T20:45:55.474 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c40a8c30 con 0x7f72c413eec0 2023-10-30T20:45:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.467+0000 7f72c99b7640 10 monclient(hunting): authenticate will time out at 2878.756461s 2023-10-30T20:45:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.468+0000 7f72c37fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c413eec0 0x7f72c413f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.468+0000 7f72c27fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f72c40b1b90 0x7f72c40a9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.468+0000 7f72c2ffd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f72c40a8850 0x7f72c4142380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.69:55288/0 (socket says 172.21.15.69:55288) 2023-10-30T20:45:55.476 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.468+0000 7f72c2ffd640 1 -- 172.21.15.69:0/3744838723 learned_addr learned my addr 172.21.15.69:0/3744838723 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:45:55.476 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.468+0000 7f72c37fe640 10 monclient(hunting): get_auth_request con 0x7f72c413eec0 auth_method 0 2023-10-30T20:45:55.476 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.468+0000 7f72c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:55.476 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.468+0000 7f72c37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:55.476 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.468+0000 7f72c37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.468+0000 7f72c27fc640 10 monclient(hunting): get_auth_request con 0x7f72c40b1b90 auth_method 0 2023-10-30T20:45:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.468+0000 7f72c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.468+0000 7f72c27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.468+0000 7f72c27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.469+0000 7f72c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.469+0000 7f72c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:55.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.469+0000 7f72c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:55.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.469+0000 7f72c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:55.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.469+0000 7f72c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:55.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.469+0000 7f72c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:55.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.470+0000 7f72c1ffb640 1 -- 172.21.15.69:0/3744838723 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417515330 0 0) 0x7f72b4001110 con 0x7f72c40a8850 2023-10-30T20:45:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.470+0000 7f72c1ffb640 10 monclient(hunting): _negotiate 2023-10-30T20:45:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.470+0000 7f72c1ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.470+0000 7f72c1ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.470+0000 7f72c1ffb640 10 monclient(hunting): my global_id is 21226 2023-10-30T20:45:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.470+0000 7f72c1ffb640 1 -- 172.21.15.69:0/3744838723 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ac003520 con 0x7f72c40a8850 2023-10-30T20:45:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.470+0000 7f72c37fe640 10 monclient(hunting): handle_auth_done global_id 18290 payload 274 2023-10-30T20:45:55.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.470+0000 7f72c37fe640 1 -- 172.21.15.69:0/3744838723 >> v2:172.21.15.69:3300/0 conn(0x7f72c40b1b90 msgr2=0x7f72c40a9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:55.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.471+0000 7f72c37fe640 1 --2- 172.21.15.69:0/3744838723 >> v2:172.21.15.69:3300/0 conn(0x7f72c40b1b90 0x7f72c40a9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.471+0000 7f72c37fe640 1 -- 172.21.15.69:0/3744838723 >> v1:172.21.15.183:6789/0 conn(0x7f72c40a8850 legacy=0x7f72c4142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:55.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.471+0000 7f72c37fe640 10 monclient: _finish_hunting 0 2023-10-30T20:45:55.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.471+0000 7f72c37fe640 1 monclient: found mon.noname-b 2023-10-30T20:45:55.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.471+0000 7f72c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:55.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.471+0000 7f72c37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:45:55.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.471+0000 7f72c37fe640 1 -- 172.21.15.69:0/3744838723 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c4142a90 con 0x7f72c413eec0 2023-10-30T20:45:55.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.472+0000 7f72c37fe640 1 --2- 172.21.15.69:0/3744838723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c413eec0 0x7f72c413f2c0 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009d90 tx=0x7f72bc02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae4e8c08945f93f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:55.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c1ffb640 1 -- 172.21.15.69:0/3744838723 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc02cbc0 con 0x7f72c413eec0 2023-10-30T20:45:55.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:55.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:55.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c1ffb640 10 monclient: dump: 2023-10-30T20:45:55.486 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:55.486 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:55.486 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.486 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.486 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:55.487 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:55.487 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:55.487 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:55.487 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:55.487 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:55.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c1ffb640 10 monclient: _finish_auth 0 2023-10-30T20:45:55.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c1ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:45:55.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:55.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c1ffb640 1 -- 172.21.15.69:0/3744838723 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72bc02cd30 con 0x7f72c413eec0 2023-10-30T20:45:55.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c99b7640 5 monclient: authenticate success, global_id 18290 2023-10-30T20:45:55.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c99b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:45:55.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:55.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c99b7640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:55.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c1ffb640 1 -- 172.21.15.69:0/3744838723 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc034660 con 0x7f72c413eec0 2023-10-30T20:45:55.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:55.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c99b7640 4 set_mon_vals no callback set 2023-10-30T20:45:55.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:45:55.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.476+0000 7f72c1ffb640 10 monclient: dump: 2023-10-30T20:45:55.490 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:55.490 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:55.490 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.490 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.491 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:55.491 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:55.491 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:55.491 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:55.491 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:55.491 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.477+0000 7f72c99b7640 10 monclient: shutdown 2023-10-30T20:45:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.477+0000 7f72c99b7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.477+0000 7f72c99b7640 1 -- 172.21.15.69:0/3744838723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c413eec0 msgr2=0x7f72c413f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.477+0000 7f72c99b7640 1 --2- 172.21.15.69:0/3744838723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c413eec0 0x7f72c413f2c0 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009d90 tx=0x7f72bc02be10 comp rx=0 tx=0).stop 2023-10-30T20:45:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.477+0000 7f72c99b7640 1 -- 172.21.15.69:0/3744838723 shutdown_connections 2023-10-30T20:45:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.477+0000 7f72c99b7640 1 --2- 172.21.15.69:0/3744838723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c413eec0 0x7f72c413f2c0 unknown :-1 s=CLOSED pgs=3206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.477+0000 7f72c99b7640 1 --2- 172.21.15.69:0/3744838723 >> v2:172.21.15.69:3300/0 conn(0x7f72c40b1b90 0x7f72c40a9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.477+0000 7f72c99b7640 1 -- 172.21.15.69:0/3744838723 >> 172.21.15.69:0/3744838723 conn(0x7f72c40b7210 msgr2=0x7f72c40b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.477+0000 7f72c99b7640 1 -- 172.21.15.69:0/3744838723 shutdown_connections 2023-10-30T20:45:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.477+0000 7f72c99b7640 1 -- 172.21.15.69:0/3744838723 wait complete. 2023-10-30T20:45:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.478+0000 7f72c99b7640 10 monclient: build_initial_monmap 2023-10-30T20:45:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.478+0000 7f72c99b7640 10 monclient: monmap: 2023-10-30T20:45:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:55.495 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.478+0000 7f72c99b7640 1 librados: starting msgr at 2023-10-30T20:45:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.478+0000 7f72c99b7640 1 librados: starting objecter 2023-10-30T20:45:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.478+0000 7f72c99b7640 1 Processor -- start 2023-10-30T20:45:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.478+0000 7f72c99b7640 1 -- start start 2023-10-30T20:45:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.478+0000 7f72c99b7640 1 librados: setting wanted keys 2023-10-30T20:45:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.478+0000 7f72c99b7640 1 librados: calling monclient init 2023-10-30T20:45:55.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.478+0000 7f72c99b7640 10 monclient: init 2023-10-30T20:45:55.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:55.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T20:45:55.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72c40a8850 0x7f72c4145ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:55.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 10 monclient(hunting): picked mon.noname-b con 0x7f72c40a8850 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:55.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c40b1b90 0x7f72c40b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 10 monclient(hunting): picked mon.noname-a con 0x7f72c40b1b90 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c413eec0 0x7f72c4142e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 10 monclient(hunting): picked mon.noname-c con 0x7f72c413eec0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c40aa3a0 con 0x7f72c40b1b90 2023-10-30T20:45:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c40b1270 con 0x7f72c40a8850 2023-10-30T20:45:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c41455a0 con 0x7f72c413eec0 2023-10-30T20:45:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c99b7640 10 monclient(hunting): authenticate will time out at 2878.768196s 2023-10-30T20:45:55.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c27fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c40b1b90 0x7f72c40b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:55.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.479+0000 7f72c37fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c413eec0 0x7f72c4142e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:55.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c2ffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72c40a8850 0x7f72c4145ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:55.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c27fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c40b1b90 0x7f72c40b0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50660/0 (socket says 172.21.15.69:50660) 2023-10-30T20:45:55.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c2ffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72c40a8850 0x7f72c4145ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:35826/0 (socket says 172.21.15.69:35826) 2023-10-30T20:45:55.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c2ffd640 1 -- 172.21.15.69:0/4028156755 learned_addr learned my addr 172.21.15.69:0/4028156755 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:55.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c2ffd640 10 monclient(hunting): get_auth_request con 0x7f72c40a8850 auth_method 0 2023-10-30T20:45:55.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:55.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:55.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:55.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c27fc640 10 monclient(hunting): get_auth_request con 0x7f72c40b1b90 auth_method 0 2023-10-30T20:45:55.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:55.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:55.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:55.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c37fe640 10 monclient(hunting): get_auth_request con 0x7f72c413eec0 auth_method 0 2023-10-30T20:45:55.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:55.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:55.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:55.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:55.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:55.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:55.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:55.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:55.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.480+0000 7f72c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:55.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.481+0000 7f72c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:55.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.481+0000 7f72c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:55.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.481+0000 7f72c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:55.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.483+0000 7f72c2ffd640 10 monclient(hunting): handle_auth_done global_id 21229 payload 824 2023-10-30T20:45:55.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.483+0000 7f72c2ffd640 1 -- 172.21.15.69:0/4028156755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c413eec0 msgr2=0x7f72c4142e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:55.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.483+0000 7f72c2ffd640 1 --2- 172.21.15.69:0/4028156755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c413eec0 0x7f72c4142e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.483+0000 7f72c2ffd640 1 -- 172.21.15.69:0/4028156755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c40b1b90 msgr2=0x7f72c40b0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:55.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.483+0000 7f72c2ffd640 1 --2- 172.21.15.69:0/4028156755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c40b1b90 0x7f72c40b0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.483+0000 7f72c2ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:45:55.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.483+0000 7f72c2ffd640 1 monclient: found mon.noname-b 2023-10-30T20:45:55.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.483+0000 7f72c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:55.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.483+0000 7f72c2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:45:55.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.483+0000 7f72c2ffd640 1 -- 172.21.15.69:0/4028156755 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c4145290 con 0x7f72c40a8850 2023-10-30T20:45:55.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.485+0000 7f72c2ffd640 1 --2- 172.21.15.69:0/4028156755 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72c40a8850 0x7f72c4145ac0 secure :-1 s=READY pgs=5048 cs=0 l=1 rev1=1 crypto rx=0x7f72b400c400 tx=0x7f72b400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:55.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.485+0000 7f72abfff640 10 client.?.objecter ms_handle_connect 0x7f72c40a8850 2023-10-30T20:45:55.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.485+0000 7f72abfff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:55.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.488+0000 7f72abfff640 1 -- 172.21.15.69:0/4028156755 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b4005ce0 con 0x7f72c40a8850 2023-10-30T20:45:55.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.488+0000 7f72abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:55.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.488+0000 7f72abfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:55.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.488+0000 7f72abfff640 10 monclient: dump: 2023-10-30T20:45:55.509 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:55.509 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:55.509 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.509 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.509 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:55.509 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:55.510 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:55.510 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:55.510 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:55.510 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:55.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.488+0000 7f72abfff640 10 monclient: _finish_auth 0 2023-10-30T20:45:55.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.488+0000 7f72abfff640 10 monclient: _check_auth_tickets 2023-10-30T20:45:55.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.488+0000 7f72abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:55.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.488+0000 7f72abfff640 1 -- 172.21.15.69:0/4028156755 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b400e040 con 0x7f72c40a8850 2023-10-30T20:45:55.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.488+0000 7f72abfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:55.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72abfff640 1 -- 172.21.15.69:0/4028156755 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b40072d0 con 0x7f72c40a8850 2023-10-30T20:45:55.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 5 monclient: authenticate success, global_id 21229 2023-10-30T20:45:55.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 10 monclient: _renew_subs 2023-10-30T20:45:55.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:55.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 1 -- 172.21.15.69:0/4028156755 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c41433c0 con 0x7f72c40a8850 2023-10-30T20:45:55.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:55.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c17fa640 4 set_mon_vals no callback set 2023-10-30T20:45:55.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:55.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72abfff640 10 monclient: dump: 2023-10-30T20:45:55.513 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:55.513 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:55.513 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.513 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.514 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:55.514 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:55.514 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:55.514 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:55.514 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:55.515 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:55.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 10 client.21229.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:55.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 10 monclient: _renew_subs 2023-10-30T20:45:55.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:55.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 1 -- 172.21.15.69:0/4028156755 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c41d6b30 con 0x7f72c40a8850 2023-10-30T20:45:55.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 1 librados: init done 2023-10-30T20:45:55.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 10 client.21229.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:55.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 15 client.21229.objecter _session_command_op_assign -1 1 2023-10-30T20:45:55.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 20 client.21229.objecter _get_session osd=-1 returning homeless 2023-10-30T20:45:55.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 10 client.21229.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:55.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:55.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.489+0000 7f72c99b7640 1 -- 172.21.15.69:0/4028156755 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f72c4141b80 con 0x7f72c40a8850 2023-10-30T20:45:55.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.509+0000 7f72abfff640 1 -- 172.21.15.69:0/4028156755 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f72b4007440 con 0x7f72c40a8850 2023-10-30T20:45:55.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.509+0000 7f72abfff640 1 --2- 172.21.15.69:0/4028156755 >> v2:172.21.15.69:6800/1876307353 conn(0x7f72b006c7c0 0x7f72b006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:55.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.509+0000 7f72c27fc640 1 --2- 172.21.15.69:0/4028156755 >> v2:172.21.15.69:6800/1876307353 conn(0x7f72b006c7c0 0x7f72b006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:55.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.509+0000 7f72c27fc640 10 monclient: get_auth_request con 0x7f72b006c7c0 auth_method 0 2023-10-30T20:45:55.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.510+0000 7f72abfff640 10 client.21229.objecter ms_handle_connect 0x7f72b006c7c0 2023-10-30T20:45:55.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.510+0000 7f72c27fc640 1 --2- 172.21.15.69:0/4028156755 >> v2:172.21.15.69:6800/1876307353 conn(0x7f72b006c7c0 0x7f72b006ec80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f72c41447e0 tx=0x7f72b8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:55.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 1 -- 172.21.15.69:0/4028156755 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f72b4057b60 con 0x7f72c40a8850 2023-10-30T20:45:55.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 10 client.21229.objecter ms_dispatch 0x7f72c4000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:45:55.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 3 client.21229.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:45:55.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 3 client.21229.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:45:55.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 10 client.21229.objecter checking command 1 2023-10-30T20:45:55.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 1 --2- 172.21.15.69:0/4028156755 >> v2:172.21.15.69:6813/431025540 conn(0x7f72b0073c00 0x7f72b00760a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:55.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 20 client.21229.objecter _get_session s=0x7f72b40072d0 osd=1 3 2023-10-30T20:45:55.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 20 client.21229.objecter put_session s=0x7f72b40072d0 osd=1 3 2023-10-30T20:45:55.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 15 client.21229.objecter _session_command_op_remove -1 1 2023-10-30T20:45:55.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 20 client.21229.objecter _get_session s=0x7f72b40072d0 osd=1 3 2023-10-30T20:45:55.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 20 client.21229.objecter get_session s=0x7f72b40072d0 osd=1 3 2023-10-30T20:45:55.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 15 client.21229.objecter _session_command_op_assign 1 1 2023-10-30T20:45:55.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 20 client.21229.objecter put_session s=0x7f72b40072d0 osd=1 4 2023-10-30T20:45:55.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 10 client.21229.objecter _send_command 1 2023-10-30T20:45:55.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 1 -- 172.21.15.69:0/4028156755 --> v2:172.21.15.69:6813/431025540 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72b4005ce0 con 0x7f72b0073c00 2023-10-30T20:45:55.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.511+0000 7f72abfff640 20 client.21229.objecter dump_active .. 0 homeless 2023-10-30T20:45:55.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.512+0000 7f72c37fe640 1 --2- 172.21.15.69:0/4028156755 >> v2:172.21.15.69:6813/431025540 conn(0x7f72b0073c00 0x7f72b00760a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:55.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.512+0000 7f72c37fe640 10 monclient: get_auth_request con 0x7f72b0073c00 auth_method 0 2023-10-30T20:45:55.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.513+0000 7f72abfff640 1 -- 172.21.15.69:0/4028156755 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=904) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72b4053ea0 con 0x7f72c40a8850 2023-10-30T20:45:55.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.513+0000 7f72abfff640 10 monclient: handle_get_version_reply finishing 1 version 904 2023-10-30T20:45:55.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.515+0000 7f72abfff640 10 client.21229.objecter ms_handle_connect 0x7f72b0073c00 2023-10-30T20:45:55.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.515+0000 7f72c37fe640 1 --2- 172.21.15.69:0/4028156755 >> v2:172.21.15.69:6813/431025540 conn(0x7f72b0073c00 0x7f72b00760a0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:55.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.537+0000 7f72abfff640 1 -- 172.21.15.69:0/4028156755 <== osd.1 v2:172.21.15.69:6813/431025540 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f72b4005ce0 con 0x7f72b0073c00 2023-10-30T20:45:55.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.537+0000 7f72abfff640 10 client.21229.objecter ms_dispatch 0x7f72c4000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:45:55.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.537+0000 7f72abfff640 10 client.21229.objecter _finish_command 1 = system:0 2023-10-30T20:45:55.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.537+0000 7f72abfff640 20 client.21229.objecter put_session s=0x7f72b40072d0 osd=1 4 2023-10-30T20:45:55.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.537+0000 7f72abfff640 15 client.21229.objecter _session_command_op_remove 1 1 2023-10-30T20:45:55.551 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.550+0000 7f72c99b7640 10 client.21229.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T20:45:55.551 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.550+0000 7f72c99b7640 15 client.21229.objecter _session_command_op_assign -1 2 2023-10-30T20:45:55.551 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.550+0000 7f72c99b7640 20 client.21229.objecter _get_session s=0x7f72b40072d0 osd=1 3 2023-10-30T20:45:55.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.550+0000 7f72c99b7640 20 client.21229.objecter put_session s=0x7f72b40072d0 osd=1 3 2023-10-30T20:45:55.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.550+0000 7f72c99b7640 20 client.21229.objecter _get_session s=0x7f72b40072d0 osd=1 3 2023-10-30T20:45:55.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.550+0000 7f72c99b7640 15 client.21229.objecter _session_command_op_remove -1 2 2023-10-30T20:45:55.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.550+0000 7f72c99b7640 20 client.21229.objecter get_session s=0x7f72b40072d0 osd=1 3 2023-10-30T20:45:55.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.550+0000 7f72c99b7640 15 client.21229.objecter _session_command_op_assign 1 2 2023-10-30T20:45:55.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.550+0000 7f72c99b7640 20 client.21229.objecter put_session s=0x7f72b40072d0 osd=1 4 2023-10-30T20:45:55.553 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.550+0000 7f72c99b7640 10 client.21229.objecter _send_command 2 2023-10-30T20:45:55.553 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.550+0000 7f72c99b7640 1 -- 172.21.15.69:0/4028156755 --> v2:172.21.15.69:6813/431025540 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f72c4141e60 con 0x7f72b0073c00 2023-10-30T20:45:55.564 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72abfff640 1 -- 172.21.15.69:0/4028156755 <== osd.1 v2:172.21.15.69:6813/431025540 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f72bc009a70 con 0x7f72b0073c00 2023-10-30T20:45:55.565 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72abfff640 10 client.21229.objecter ms_dispatch 0x7f72c4000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:45:55.565 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72abfff640 10 client.21229.objecter _finish_command 2 = system:0 2023-10-30T20:45:55.565 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72abfff640 20 client.21229.objecter put_session s=0x7f72b40072d0 osd=1 4 2023-10-30T20:45:55.565 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72abfff640 15 client.21229.objecter _session_command_op_remove 1 2 2023-10-30T20:45:55.565 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-10-30T20:45:55.566 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T20:45:55.566 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-10-30T20:45:55.567 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72c99b7640 10 librados: watch_flush enter 2023-10-30T20:45:55.567 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72c99b7640 10 librados: watch_flush exit 2023-10-30T20:45:55.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72c99b7640 10 client.21229.objecter close_session for osd.1 2023-10-30T20:45:55.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72c99b7640 1 -- 172.21.15.69:0/4028156755 >> v2:172.21.15.69:6813/431025540 conn(0x7f72b0073c00 msgr2=0x7f72b00760a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:55.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72c99b7640 1 --2- 172.21.15.69:0/4028156755 >> v2:172.21.15.69:6813/431025540 conn(0x7f72b0073c00 0x7f72b00760a0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72c99b7640 20 client.21229.objecter put_session s=0x7f72b40072d0 osd=1 1 2023-10-30T20:45:55.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72c99b7640 20 client.21229.objecter shutdown clearing up homeless session... 2023-10-30T20:45:55.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72c99b7640 10 client.21229.objecter successfully canceled tick 2023-10-30T20:45:55.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72c99b7640 1 -- 172.21.15.69:0/4028156755 >> v2:172.21.15.69:6800/1876307353 conn(0x7f72b006c7c0 msgr2=0x7f72b006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:55.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72c99b7640 1 --2- 172.21.15.69:0/4028156755 >> v2:172.21.15.69:6800/1876307353 conn(0x7f72b006c7c0 0x7f72b006ec80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f72c41447e0 tx=0x7f72b8009190 comp rx=0 tx=0).stop 2023-10-30T20:45:55.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72c99b7640 10 monclient: shutdown 2023-10-30T20:45:55.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72c99b7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:55.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72c99b7640 1 -- 172.21.15.69:0/4028156755 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72c40a8850 msgr2=0x7f72c4145ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:55.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.563+0000 7f72c99b7640 1 --2- 172.21.15.69:0/4028156755 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72c40a8850 0x7f72c4145ac0 secure :-1 s=READY pgs=5048 cs=0 l=1 rev1=1 crypto rx=0x7f72b400c400 tx=0x7f72b400c8c0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.564+0000 7f72c37fe640 1 -- 172.21.15.69:0/4028156755 reap_dead start 2023-10-30T20:45:55.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.564+0000 7f72c99b7640 1 -- 172.21.15.69:0/4028156755 shutdown_connections 2023-10-30T20:45:55.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.564+0000 7f72c99b7640 1 -- 172.21.15.69:0/4028156755 >> 172.21.15.69:0/4028156755 conn(0x7f72c40b7210 msgr2=0x7f72c4138860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:55.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.564+0000 7f72c99b7640 1 -- 172.21.15.69:0/4028156755 shutdown_connections 2023-10-30T20:45:55.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.564+0000 7f72c99b7640 1 -- 172.21.15.69:0/4028156755 wait complete. 2023-10-30T20:45:55.571 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.564+0000 7f72c99b7640 1 librados: shutdown 2023-10-30T20:45:55.575 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-30T20:45:55.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.676+0000 7f951ed05640 10 monclient: get_monmap_and_config 2023-10-30T20:45:55.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.676+0000 7f951ed05640 10 monclient: build_initial_monmap 2023-10-30T20:45:55.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.676+0000 7f951ed05640 10 monclient: monmap: 2023-10-30T20:45:55.678 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:55.679 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:55.679 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:55.677853+0000 2023-10-30T20:45:55.679 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:55.677853+0000 2023-10-30T20:45:55.679 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:55.679 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:55.679 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:55.680 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:55.680 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:55.680 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:55.680 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:55.680 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:55.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.677+0000 7f951ed05640 1 Processor -- start 2023-10-30T20:45:55.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.677+0000 7f951ed05640 1 -- start start 2023-10-30T20:45:55.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.677+0000 7f951ed05640 10 monclient: init 2023-10-30T20:45:55.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:55.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-10-30T20:45:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95180a8e40 0x7f951813efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 10 monclient(hunting): picked mon.noname-b con 0x7f95180a8e40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f95180b13c0 0x7f9518141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 10 monclient(hunting): picked mon.noname-c con 0x7f95180b13c0 addr v2:172.21.15.183:3300/0 2023-10-30T20:45:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f951813f500 0x7f95181423f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:55.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 10 monclient(hunting): picked mon.noname-a con 0x7f951813f500 addr v2:172.21.15.69:3300/0 2023-10-30T20:45:55.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:55.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95180679a0 con 0x7f951813f500 2023-10-30T20:45:55.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:55.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95180b1060 con 0x7f95180b13c0 2023-10-30T20:45:55.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:55.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95180a9220 con 0x7f95180a8e40 2023-10-30T20:45:55.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:55.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.678+0000 7f951ed05640 10 monclient(hunting): authenticate will time out at 2878.967196s 2023-10-30T20:45:55.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f951d2a2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f951813f500 0x7f95181423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:55.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f9517fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f95180b13c0 0x7f9518141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:55.687 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f951caa1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95180a8e40 0x7f951813efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:55.687 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f951d2a2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f951813f500 0x7f95181423f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50664/0 (socket says 172.21.15.69:50664) 2023-10-30T20:45:55.687 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f951d2a2640 1 -- 172.21.15.69:0/397542710 learned_addr learned my addr 172.21.15.69:0/397542710 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:55.687 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f951d2a2640 10 monclient(hunting): get_auth_request con 0x7f951813f500 auth_method 0 2023-10-30T20:45:55.687 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f951d2a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:55.687 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f951d2a2640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:55.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f951d2a2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:55.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f9517fff640 10 monclient(hunting): get_auth_request con 0x7f95180b13c0 auth_method 0 2023-10-30T20:45:55.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f9517fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:55.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f9517fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:55.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f9517fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:55.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f951caa1640 10 monclient(hunting): get_auth_request con 0x7f95180a8e40 auth_method 0 2023-10-30T20:45:55.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f951caa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:55.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f951caa1640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:55.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.679+0000 7f951caa1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:55.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.680+0000 7f951d2a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:55.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.680+0000 7f951d2a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:55.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.680+0000 7f951d2a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:55.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.680+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:55.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.680+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:55.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.680+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:55.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.680+0000 7f951caa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:55.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.680+0000 7f951caa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:55.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.680+0000 7f951caa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:55.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.682+0000 7f951caa1640 10 monclient(hunting): handle_auth_done global_id 18296 payload 274 2023-10-30T20:45:55.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.682+0000 7f951caa1640 1 -- 172.21.15.69:0/397542710 >> v2:172.21.15.183:3300/0 conn(0x7f95180b13c0 msgr2=0x7f9518141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:55.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.682+0000 7f951caa1640 1 --2- 172.21.15.69:0/397542710 >> v2:172.21.15.183:3300/0 conn(0x7f95180b13c0 0x7f9518141eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.682+0000 7f951caa1640 1 -- 172.21.15.69:0/397542710 >> v2:172.21.15.69:3300/0 conn(0x7f951813f500 msgr2=0x7f95181423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:55.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.682+0000 7f951caa1640 1 --2- 172.21.15.69:0/397542710 >> v2:172.21.15.69:3300/0 conn(0x7f951813f500 0x7f95181423f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.682+0000 7f951caa1640 10 monclient: _finish_hunting 0 2023-10-30T20:45:55.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.682+0000 7f951caa1640 1 monclient: found mon.noname-b 2023-10-30T20:45:55.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.682+0000 7f951caa1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:55.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.682+0000 7f951caa1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:45:55.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.682+0000 7f951caa1640 1 -- 172.21.15.69:0/397542710 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95180a8920 con 0x7f95180a8e40 2023-10-30T20:45:55.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.684+0000 7f951caa1640 1 --2- 172.21.15.69:0/397542710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95180a8e40 0x7f951813efc0 secure :-1 s=READY pgs=3207 cs=0 l=1 rev1=1 crypto rx=0x7f95180721d0 tx=0x7f950c02b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98bb5dbe0723bb21 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:55.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f95177fe640 1 -- 172.21.15.69:0/397542710 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c02e050 con 0x7f95180a8e40 2023-10-30T20:45:55.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f95177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:55.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f95177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:55.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f95177fe640 10 monclient: dump: 2023-10-30T20:45:55.697 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:55.697 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:55.697 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.697 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.697 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:55.697 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:55.698 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:55.698 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:55.698 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:55.698 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:55.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f95177fe640 10 monclient: _finish_auth 0 2023-10-30T20:45:55.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f95177fe640 10 monclient: _check_auth_tickets 2023-10-30T20:45:55.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f95177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:55.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f95177fe640 1 -- 172.21.15.69:0/397542710 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950c030710 con 0x7f95180a8e40 2023-10-30T20:45:55.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f95177fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:55.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f95177fe640 1 -- 172.21.15.69:0/397542710 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c030ab0 con 0x7f95180a8e40 2023-10-30T20:45:55.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f95177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:55.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f95177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:45:55.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f95177fe640 10 monclient: dump: 2023-10-30T20:45:55.700 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:55.700 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:55.700 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.701 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.701 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:55.701 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:55.701 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:55.701 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:55.701 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:55.702 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:55.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f951ed05640 5 monclient: authenticate success, global_id 18296 2023-10-30T20:45:55.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f951ed05640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:55.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.688+0000 7f951ed05640 4 set_mon_vals no callback set 2023-10-30T20:45:55.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.689+0000 7f951ed05640 10 monclient: shutdown 2023-10-30T20:45:55.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.689+0000 7f951ed05640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:55.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.689+0000 7f951ed05640 1 -- 172.21.15.69:0/397542710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95180a8e40 msgr2=0x7f951813efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:55.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.689+0000 7f951ed05640 1 --2- 172.21.15.69:0/397542710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95180a8e40 0x7f951813efc0 secure :-1 s=READY pgs=3207 cs=0 l=1 rev1=1 crypto rx=0x7f95180721d0 tx=0x7f950c02b710 comp rx=0 tx=0).stop 2023-10-30T20:45:55.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.689+0000 7f951ed05640 1 -- 172.21.15.69:0/397542710 shutdown_connections 2023-10-30T20:45:55.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.689+0000 7f951ed05640 1 --2- 172.21.15.69:0/397542710 >> v2:172.21.15.69:3300/0 conn(0x7f951813f500 0x7f95181423f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.689+0000 7f951ed05640 1 --2- 172.21.15.69:0/397542710 >> v2:172.21.15.183:3300/0 conn(0x7f95180b13c0 0x7f9518141eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.689+0000 7f951ed05640 1 --2- 172.21.15.69:0/397542710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95180a8e40 0x7f951813efc0 unknown :-1 s=CLOSED pgs=3207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.689+0000 7f951ed05640 1 -- 172.21.15.69:0/397542710 >> 172.21.15.69:0/397542710 conn(0x7f95180b7180 msgr2=0x7f95180b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:55.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.689+0000 7f951ed05640 1 -- 172.21.15.69:0/397542710 shutdown_connections 2023-10-30T20:45:55.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.689+0000 7f951ed05640 1 -- 172.21.15.69:0/397542710 wait complete. 2023-10-30T20:45:55.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.690+0000 7f951ed05640 10 monclient: build_initial_monmap 2023-10-30T20:45:55.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.690+0000 7f951ed05640 10 monclient: monmap: 2023-10-30T20:45:55.705 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:55.705 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:55.705 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:55.705 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:55.706 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:55.706 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:55.706 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:55.706 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:55.706 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:55.706 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:55.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.690+0000 7f951ed05640 1 librados: starting msgr at 2023-10-30T20:45:55.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.690+0000 7f951ed05640 1 librados: starting objecter 2023-10-30T20:45:55.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.690+0000 7f951ed05640 1 Processor -- start 2023-10-30T20:45:55.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.690+0000 7f951ed05640 1 -- start start 2023-10-30T20:45:55.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.690+0000 7f951ed05640 1 librados: setting wanted keys 2023-10-30T20:45:55.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.690+0000 7f951ed05640 1 librados: calling monclient init 2023-10-30T20:45:55.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.690+0000 7f951ed05640 10 monclient: init 2023-10-30T20:45:55.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:55.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:45:55.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95180a8e40 0x7f95181430f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:55.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 10 monclient(hunting): picked mon.noname-a con 0x7f95180a8e40 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:55.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9518144dd0 0x7f95181da630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:55.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 10 monclient(hunting): picked mon.noname-b con 0x7f9518144dd0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:55.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95180ad9e0 0x7f95181dab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:55.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 10 monclient(hunting): picked mon.noname-c con 0x7f95180ad9e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:55.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:55.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9518142690 con 0x7f95180a8e40 2023-10-30T20:45:55.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:55.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9518144740 con 0x7f9518144dd0 2023-10-30T20:45:55.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:55.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95181449b0 con 0x7f95180ad9e0 2023-10-30T20:45:55.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:55.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.691+0000 7f951ed05640 10 monclient(hunting): authenticate will time out at 2878.980469s 2023-10-30T20:45:55.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.692+0000 7f951d2a2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95180ad9e0 0x7f95181dab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:55.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.692+0000 7f951caa1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95180a8e40 0x7f95181430f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:55.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.692+0000 7f951d2a2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95180ad9e0 0x7f95181dab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41752/0 (socket says 172.21.15.69:41752) 2023-10-30T20:45:55.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.692+0000 7f951d2a2640 1 -- 172.21.15.69:0/1975219897 learned_addr learned my addr 172.21.15.69:0/1975219897 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:55.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.692+0000 7f951d2a2640 10 monclient(hunting): get_auth_request con 0x7f95180ad9e0 auth_method 0 2023-10-30T20:45:55.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.692+0000 7f951d2a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:55.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.692+0000 7f951d2a2640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:55.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.692+0000 7f951d2a2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:55.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.692+0000 7f9517fff640 1 --2- 172.21.15.69:0/1975219897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9518144dd0 0x7f95181da630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:55.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.692+0000 7f951caa1640 10 monclient(hunting): get_auth_request con 0x7f95180a8e40 auth_method 0 2023-10-30T20:45:55.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.692+0000 7f951caa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:55.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.692+0000 7f951caa1640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:55.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.692+0000 7f951caa1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:55.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.693+0000 7f9517fff640 10 monclient(hunting): get_auth_request con 0x7f9518144dd0 auth_method 0 2023-10-30T20:45:55.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.693+0000 7f9517fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:55.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.693+0000 7f9517fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:55.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.693+0000 7f9517fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:55.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.693+0000 7f951d2a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:55.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.693+0000 7f951d2a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:55.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.693+0000 7f951d2a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:55.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.693+0000 7f951caa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:55.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.693+0000 7f951caa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:55.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.693+0000 7f951caa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:55.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.693+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:55.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.693+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:55.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.693+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:55.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.696+0000 7f951d2a2640 10 monclient(hunting): handle_auth_done global_id 18299 payload 824 2023-10-30T20:45:55.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.696+0000 7f951d2a2640 1 -- 172.21.15.69:0/1975219897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9518144dd0 msgr2=0x7f95181da630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:55.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.696+0000 7f951d2a2640 1 --2- 172.21.15.69:0/1975219897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9518144dd0 0x7f95181da630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.696+0000 7f951d2a2640 1 -- 172.21.15.69:0/1975219897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95180a8e40 msgr2=0x7f95181430f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:55.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.696+0000 7f951d2a2640 1 --2- 172.21.15.69:0/1975219897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95180a8e40 0x7f95181430f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:55.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.696+0000 7f951d2a2640 10 monclient: _finish_hunting 0 2023-10-30T20:45:55.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.696+0000 7f951d2a2640 1 monclient: found mon.noname-c 2023-10-30T20:45:55.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.696+0000 7f951d2a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:55.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.696+0000 7f951d2a2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:45:55.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.696+0000 7f951d2a2640 1 -- 172.21.15.69:0/1975219897 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95181da110 con 0x7f95180ad9e0 2023-10-30T20:45:55.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.699+0000 7f951d2a2640 1 --2- 172.21.15.69:0/1975219897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95180ad9e0 0x7f95181dab70 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7f9510009cb0 tx=0x7f951000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:55.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.699+0000 7f95157fa640 10 client.?.objecter ms_handle_connect 0x7f95180ad9e0 2023-10-30T20:45:55.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.699+0000 7f95157fa640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:55.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f95157fa640 1 -- 172.21.15.69:0/1975219897 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951000f040 con 0x7f95180ad9e0 2023-10-30T20:45:55.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f95157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:55.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f95157fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:45:55.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f95157fa640 10 monclient: dump: 2023-10-30T20:45:55.720 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:55.720 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:55.720 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.720 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.721 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:55.721 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:55.721 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:55.721 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:55.721 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:55.722 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:55.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f95157fa640 10 monclient: _finish_auth 0 2023-10-30T20:45:55.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f95157fa640 10 monclient: _check_auth_tickets 2023-10-30T20:45:55.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f95157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:55.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f951ed05640 5 monclient: authenticate success, global_id 18299 2023-10-30T20:45:55.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f951ed05640 10 monclient: _renew_subs 2023-10-30T20:45:55.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f951ed05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:45:55.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f951ed05640 1 -- 172.21.15.69:0/1975219897 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95180ac8c0 con 0x7f95180ad9e0 2023-10-30T20:45:55.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f951ed05640 10 client.18299.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:55.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f951ed05640 10 monclient: _renew_subs 2023-10-30T20:45:55.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f951ed05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:45:55.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f951ed05640 1 -- 172.21.15.69:0/1975219897 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9518145660 con 0x7f95180ad9e0 2023-10-30T20:45:55.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.702+0000 7f951ed05640 1 librados: init done 2023-10-30T20:45:55.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.703+0000 7f951ed05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:55.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.703+0000 7f951ed05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:55.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.703+0000 7f951ed05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:45:55.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.703+0000 7f951ed05640 1 -- 172.21.15.69:0/1975219897 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95180b1b00 con 0x7f95180ad9e0 2023-10-30T20:45:55.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.913+0000 7f95157fa640 1 -- 172.21.15.69:0/1975219897 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951000cd20 con 0x7f95180ad9e0 2023-10-30T20:45:55.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.913+0000 7f95157fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:55.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.913+0000 7f9516ffd640 4 set_mon_vals no callback set 2023-10-30T20:45:55.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.947+0000 7f95157fa640 1 -- 172.21.15.69:0/1975219897 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510003b90 con 0x7f95180ad9e0 2023-10-30T20:45:55.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.947+0000 7f95157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:55.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.947+0000 7f95157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:45:55.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.947+0000 7f95157fa640 10 monclient: dump: 2023-10-30T20:45:55.949 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:55.949 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:55.949 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.950 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:55.950 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:55.950 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:55.950 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:55.950 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:55.951 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:55.951 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:55.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.948+0000 7f95157fa640 1 -- 172.21.15.69:0/1975219897 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f9510004030 con 0x7f95180ad9e0 2023-10-30T20:45:55.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.948+0000 7f95157fa640 1 --2- 172.21.15.69:0/1975219897 >> v2:172.21.15.69:6800/1876307353 conn(0x7f94fc06c7f0 0x7f94fc06ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:55.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.948+0000 7f951caa1640 1 --2- 172.21.15.69:0/1975219897 >> v2:172.21.15.69:6800/1876307353 conn(0x7f94fc06c7f0 0x7f94fc06ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:55.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.948+0000 7f951caa1640 10 monclient: get_auth_request con 0x7f94fc06c7f0 auth_method 0 2023-10-30T20:45:55.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.949+0000 7f951caa1640 1 --2- 172.21.15.69:0/1975219897 >> v2:172.21.15.69:6800/1876307353 conn(0x7f94fc06c7f0 0x7f94fc06ecb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f950c034000 tx=0x7f950c02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:55.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.949+0000 7f95157fa640 10 client.18299.objecter ms_handle_connect 0x7f94fc06c7f0 2023-10-30T20:45:55.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.960+0000 7f95157fa640 1 -- 172.21.15.69:0/1975219897 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f95100591d0 con 0x7f95180ad9e0 2023-10-30T20:45:55.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.960+0000 7f95157fa640 10 client.18299.objecter ms_dispatch 0x7f9518000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:45:55.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.960+0000 7f95157fa640 3 client.18299.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:45:55.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.960+0000 7f95157fa640 3 client.18299.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:45:55.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.960+0000 7f95157fa640 20 client.18299.objecter dump_active .. 0 homeless 2023-10-30T20:45:55.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.960+0000 7f95157fa640 1 -- 172.21.15.69:0/1975219897 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9510004970 con 0x7f95180ad9e0 2023-10-30T20:45:55.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.960+0000 7f95157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:55.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:55.960+0000 7f95157fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:45:56.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.132+0000 7f951ed05640 1 -- 172.21.15.69:0/1975219897 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9518142930 con 0x7f94fc06c7f0 2023-10-30T20:45:56.134 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.132+0000 7f95157fa640 1 -- 172.21.15.69:0/1975219897 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f950c040070 con 0x7f94fc06c7f0 2023-10-30T20:45:56.134 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 1 to deep-scrub 2023-10-30T20:45:56.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.134+0000 7f951ed05640 10 librados: watch_flush enter 2023-10-30T20:45:56.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 10 librados: watch_flush exit 2023-10-30T20:45:56.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 20 client.18299.objecter shutdown clearing up homeless session... 2023-10-30T20:45:56.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 10 client.18299.objecter successfully canceled tick 2023-10-30T20:45:56.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 1 -- 172.21.15.69:0/1975219897 >> v2:172.21.15.69:6800/1876307353 conn(0x7f94fc06c7f0 msgr2=0x7f94fc06ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 1 --2- 172.21.15.69:0/1975219897 >> v2:172.21.15.69:6800/1876307353 conn(0x7f94fc06c7f0 0x7f94fc06ecb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f950c034000 tx=0x7f950c02f000 comp rx=0 tx=0).stop 2023-10-30T20:45:56.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 10 monclient: shutdown 2023-10-30T20:45:56.138 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:56.138 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 1 -- 172.21.15.69:0/1975219897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95180ad9e0 msgr2=0x7f95181dab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.138 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 1 --2- 172.21.15.69:0/1975219897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95180ad9e0 0x7f95181dab70 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7f9510009cb0 tx=0x7f951000c490 comp rx=0 tx=0).stop 2023-10-30T20:45:56.138 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 1 -- 172.21.15.69:0/1975219897 shutdown_connections 2023-10-30T20:45:56.138 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 1 --2- 172.21.15.69:0/1975219897 >> v2:172.21.15.69:6800/1876307353 conn(0x7f94fc06c7f0 0x7f94fc06ecb0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.139 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 1 --2- 172.21.15.69:0/1975219897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95180ad9e0 0x7f95181dab70 unknown :-1 s=CLOSED pgs=3208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.139 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 1 --2- 172.21.15.69:0/1975219897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9518144dd0 0x7f95181da630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.139 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 1 --2- 172.21.15.69:0/1975219897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95180a8e40 0x7f95181430f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.139 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 1 -- 172.21.15.69:0/1975219897 >> 172.21.15.69:0/1975219897 conn(0x7f95180b7180 msgr2=0x7f951813ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:56.139 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 1 -- 172.21.15.69:0/1975219897 shutdown_connections 2023-10-30T20:45:56.139 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 1 -- 172.21.15.69:0/1975219897 wait complete. 2023-10-30T20:45:56.140 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.135+0000 7f951ed05640 1 librados: shutdown 2023-10-30T20:45:56.148 INFO:tasks.ceph:Scrubbing osd.2 2023-10-30T20:45:56.148 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-30T20:45:56.251 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.250+0000 7f048ba15640 10 monclient: get_monmap_and_config 2023-10-30T20:45:56.251 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.250+0000 7f048ba15640 10 monclient: build_initial_monmap 2023-10-30T20:45:56.251 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.250+0000 7f048ba15640 10 monclient: monmap: 2023-10-30T20:45:56.251 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:56.252 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:56.252 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:56.251152+0000 2023-10-30T20:45:56.252 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:56.251152+0000 2023-10-30T20:45:56.252 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:56.252 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:56.253 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:56.253 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:56.253 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:56.253 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:56.253 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:56.254 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:56.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.250+0000 7f048ba15640 1 Processor -- start 2023-10-30T20:45:56.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.250+0000 7f048ba15640 1 -- start start 2023-10-30T20:45:56.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.250+0000 7f048ba15640 10 monclient: init 2023-10-30T20:45:56.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.251+0000 7f048ba15640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:56.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.251+0000 7f048ba15640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-10-30T20:45:56.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.251+0000 7f048ba15640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04840b4360 addr v1:172.21.15.69:6789/0 2023-10-30T20:45:56.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.251+0000 7f048ba15640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f048413f320 0x7f0484144e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:56.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.251+0000 7f048ba15640 10 monclient(hunting): picked mon.noname-a con 0x7f048413f320 addr v2:172.21.15.69:3300/0 2023-10-30T20:45:56.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.251+0000 7f048ba15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04840b12b0 0x7f0484145350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:56.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.251+0000 7f048ba15640 10 monclient(hunting): picked mon.noname-b con 0x7f04840b12b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:56.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.251+0000 7f048ba15640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04840aa180 con 0x7f04840b4360 2023-10-30T20:45:56.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.251+0000 7f048ba15640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:56.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.251+0000 7f048ba15640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0484067a30 con 0x7f048413f320 2023-10-30T20:45:56.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.251+0000 7f048ba15640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:56.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.251+0000 7f048ba15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04840b4740 con 0x7f04840b12b0 2023-10-30T20:45:56.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.251+0000 7f048ba15640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:56.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.251+0000 7f048ba15640 10 monclient(hunting): authenticate will time out at 2879.540342s 2023-10-30T20:45:56.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.252+0000 7f0489fb2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04840b12b0 0x7f0484145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:56.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.252+0000 7f0488fb0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f048413f320 0x7f0484144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:56.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.252+0000 7f0489fb2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04840b12b0 0x7f0484145350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41766/0 (socket says 172.21.15.69:41766) 2023-10-30T20:45:56.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.252+0000 7f0489fb2640 1 -- 172.21.15.69:0/3215952039 learned_addr learned my addr 172.21.15.69:0/3215952039 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:56.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.252+0000 7f0489fb2640 10 monclient(hunting): get_auth_request con 0x7f04840b12b0 auth_method 0 2023-10-30T20:45:56.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.252+0000 7f0489fb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:56.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.252+0000 7f0489fb2640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:56.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.252+0000 7f0489fb2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:56.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.252+0000 7f0488fb0640 10 monclient(hunting): get_auth_request con 0x7f048413f320 auth_method 0 2023-10-30T20:45:56.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.252+0000 7f0488fb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:56.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.252+0000 7f0488fb0640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:56.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.252+0000 7f0488fb0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:56.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.253+0000 7f0489fb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:56.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.253+0000 7f0489fb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:56.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.253+0000 7f0489fb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:56.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.253+0000 7f0488fb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:56.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.253+0000 7f0488fb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:56.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.253+0000 7f0488fb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:56.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.255+0000 7f0489fb2640 10 monclient(hunting): handle_auth_done global_id 18302 payload 274 2023-10-30T20:45:56.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.255+0000 7f0489fb2640 1 -- 172.21.15.69:0/3215952039 >> v2:172.21.15.69:3300/0 conn(0x7f048413f320 msgr2=0x7f0484144e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:56.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.255+0000 7f0489fb2640 1 --2- 172.21.15.69:0/3215952039 >> v2:172.21.15.69:3300/0 conn(0x7f048413f320 0x7f0484144e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.255+0000 7f0489fb2640 1 -- 172.21.15.69:0/3215952039 >> v1:172.21.15.69:6789/0 conn(0x7f04840b4360 legacy=0x7f048413ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.255+0000 7f0489fb2640 10 monclient: _finish_hunting 0 2023-10-30T20:45:56.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.255+0000 7f0489fb2640 1 monclient: found mon.noname-b 2023-10-30T20:45:56.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.255+0000 7f0489fb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:56.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.255+0000 7f0489fb2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:45:56.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.255+0000 7f0489fb2640 1 -- 172.21.15.69:0/3215952039 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04840aa3d0 con 0x7f04840b12b0 2023-10-30T20:45:56.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.257+0000 7f0489fb2640 1 --2- 172.21.15.69:0/3215952039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04840b12b0 0x7f0484145350 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f047c009e50 tx=0x7f047c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22e54dda6d1fb61b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:56.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.260+0000 7f0477fff640 1 -- 172.21.15.69:0/3215952039 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c02ca70 con 0x7f04840b12b0 2023-10-30T20:45:56.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f0477fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:56.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f0477fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:56.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f0477fff640 10 monclient: dump: 2023-10-30T20:45:56.268 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:56.268 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:56.268 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.268 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.268 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:56.269 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:56.269 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:56.269 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:56.269 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:56.269 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:56.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f0477fff640 10 monclient: _finish_auth 0 2023-10-30T20:45:56.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f0477fff640 10 monclient: _check_auth_tickets 2023-10-30T20:45:56.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f0477fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:56.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f0477fff640 1 -- 172.21.15.69:0/3215952039 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c02cbe0 con 0x7f04840b12b0 2023-10-30T20:45:56.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f0477fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:56.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f0477fff640 1 -- 172.21.15.69:0/3215952039 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c0343f0 con 0x7f04840b12b0 2023-10-30T20:45:56.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f048ba15640 5 monclient: authenticate success, global_id 18302 2023-10-30T20:45:56.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f048ba15640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:56.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f0477fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:56.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f048ba15640 4 set_mon_vals no callback set 2023-10-30T20:45:56.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f0477fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:45:56.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f0477fff640 10 monclient: dump: 2023-10-30T20:45:56.272 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:56.272 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:56.272 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.272 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.272 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:56.272 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:56.273 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:56.273 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:56.273 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:56.273 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:56.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f048ba15640 10 monclient: shutdown 2023-10-30T20:45:56.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f048ba15640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:56.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f048ba15640 1 -- 172.21.15.69:0/3215952039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04840b12b0 msgr2=0x7f0484145350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f048ba15640 1 --2- 172.21.15.69:0/3215952039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04840b12b0 0x7f0484145350 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f047c009e50 tx=0x7f047c02bed0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f048ba15640 1 -- 172.21.15.69:0/3215952039 shutdown_connections 2023-10-30T20:45:56.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f048ba15640 1 --2- 172.21.15.69:0/3215952039 >> v2:172.21.15.69:3300/0 conn(0x7f048413f320 0x7f0484144e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f048ba15640 1 --2- 172.21.15.69:0/3215952039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04840b12b0 0x7f0484145350 unknown :-1 s=CLOSED pgs=3209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f048ba15640 1 -- 172.21.15.69:0/3215952039 >> 172.21.15.69:0/3215952039 conn(0x7f0484065ed0 msgr2=0x7f04840662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:56.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f048ba15640 1 -- 172.21.15.69:0/3215952039 shutdown_connections 2023-10-30T20:45:56.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.261+0000 7f048ba15640 1 -- 172.21.15.69:0/3215952039 wait complete. 2023-10-30T20:45:56.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.262+0000 7f048ba15640 10 monclient: build_initial_monmap 2023-10-30T20:45:56.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.262+0000 7f048ba15640 10 monclient: monmap: 2023-10-30T20:45:56.276 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:56.276 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:56.276 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:56.276 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:56.276 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:56.276 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:56.277 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:56.277 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:56.277 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:56.277 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:56.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.262+0000 7f048ba15640 1 librados: starting msgr at 2023-10-30T20:45:56.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.262+0000 7f048ba15640 1 librados: starting objecter 2023-10-30T20:45:56.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.262+0000 7f048ba15640 1 Processor -- start 2023-10-30T20:45:56.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.262+0000 7f048ba15640 1 -- start start 2023-10-30T20:45:56.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.262+0000 7f048ba15640 1 librados: setting wanted keys 2023-10-30T20:45:56.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.262+0000 7f048ba15640 1 librados: calling monclient init 2023-10-30T20:45:56.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.262+0000 7f048ba15640 10 monclient: init 2023-10-30T20:45:56.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:56.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T20:45:56.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04840b12b0 0x7f0484140620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:56.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 10 monclient(hunting): picked mon.noname-c con 0x7f04840b12b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:56.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f04840b4360 0x7f04841d6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:56.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 10 monclient(hunting): picked mon.noname-b con 0x7f04840b4360 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:56.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0484141010 0x7f04841d7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:56.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 10 monclient(hunting): picked mon.noname-a con 0x7f0484141010 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:56.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:56.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048413eed0 con 0x7f0484141010 2023-10-30T20:45:56.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:56.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04841d7590 con 0x7f04840b4360 2023-10-30T20:45:56.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:56.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04841d65f0 con 0x7f04840b12b0 2023-10-30T20:45:56.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:56.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f048ba15640 10 monclient(hunting): authenticate will time out at 2879.552336s 2023-10-30T20:45:56.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f04897b1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04840b12b0 0x7f0484140620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:56.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.263+0000 7f0489fb2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0484141010 0x7f04841d7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:56.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0488fb0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f04840b4360 0x7f04841d6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:56.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f04897b1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04840b12b0 0x7f0484140620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41780/0 (socket says 172.21.15.69:41780) 2023-10-30T20:45:56.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0489fb2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0484141010 0x7f04841d7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50686/0 (socket says 172.21.15.69:50686) 2023-10-30T20:45:56.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f04897b1640 1 -- 172.21.15.69:0/1079505482 learned_addr learned my addr 172.21.15.69:0/1079505482 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:56.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f04897b1640 10 monclient(hunting): get_auth_request con 0x7f04840b12b0 auth_method 0 2023-10-30T20:45:56.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f04897b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:56.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f04897b1640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:56.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f04897b1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:56.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0489fb2640 10 monclient(hunting): get_auth_request con 0x7f0484141010 auth_method 0 2023-10-30T20:45:56.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0489fb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:56.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0489fb2640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:56.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0489fb2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:56.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0488fb0640 10 monclient(hunting): get_auth_request con 0x7f04840b4360 auth_method 0 2023-10-30T20:45:56.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0488fb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:56.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0488fb0640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:56.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0488fb0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:56.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f04897b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:56.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f04897b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:56.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f04897b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:56.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0488fb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:56.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0488fb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:56.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0488fb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:56.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0489fb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:56.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0489fb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:56.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.264+0000 7f0489fb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:56.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.267+0000 7f0489fb2640 10 monclient(hunting): handle_auth_done global_id 16269 payload 824 2023-10-30T20:45:56.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.267+0000 7f0489fb2640 1 -- 172.21.15.69:0/1079505482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04840b12b0 msgr2=0x7f0484140620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.267+0000 7f0489fb2640 1 --2- 172.21.15.69:0/1079505482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04840b12b0 0x7f0484140620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.267+0000 7f0489fb2640 1 -- 172.21.15.69:0/1079505482 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f04840b4360 msgr2=0x7f04841d6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.267+0000 7f0489fb2640 1 --2- 172.21.15.69:0/1079505482 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f04840b4360 0x7f04841d6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.267+0000 7f0489fb2640 10 monclient: _finish_hunting 0 2023-10-30T20:45:56.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.267+0000 7f0489fb2640 1 monclient: found mon.noname-a 2023-10-30T20:45:56.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.267+0000 7f0489fb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:56.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.267+0000 7f0489fb2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:45:56.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.267+0000 7f0489fb2640 1 -- 172.21.15.69:0/1079505482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0484142c10 con 0x7f0484141010 2023-10-30T20:45:56.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.269+0000 7f0489fb2640 1 --2- 172.21.15.69:0/1079505482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0484141010 0x7f04841d7050 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 crypto rx=0x7f04840aab80 tx=0x7f047c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:56.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.269+0000 7f0475ffb640 10 client.?.objecter ms_handle_connect 0x7f0484141010 2023-10-30T20:45:56.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.269+0000 7f0475ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:56.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f0475ffb640 1 -- 172.21.15.69:0/1079505482 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c009510 con 0x7f0484141010 2023-10-30T20:45:56.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f0475ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:56.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f0475ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:45:56.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f0475ffb640 10 monclient: dump: 2023-10-30T20:45:56.291 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:56.291 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:56.291 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.291 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.292 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:56.292 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:56.292 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:56.292 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:56.292 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:56.292 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:56.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f0475ffb640 10 monclient: _finish_auth 0 2023-10-30T20:45:56.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f0475ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:45:56.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f0475ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:56.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f0475ffb640 1 -- 172.21.15.69:0/1079505482 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c03e450 con 0x7f0484141010 2023-10-30T20:45:56.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f0475ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:56.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f0475ffb640 1 -- 172.21.15.69:0/1079505482 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c03e830 con 0x7f0484141010 2023-10-30T20:45:56.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f048ba15640 5 monclient: authenticate success, global_id 16269 2023-10-30T20:45:56.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f048ba15640 10 monclient: _renew_subs 2023-10-30T20:45:56.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f048ba15640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:45:56.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f048ba15640 1 -- 172.21.15.69:0/1079505482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0484144620 con 0x7f0484141010 2023-10-30T20:45:56.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f04777fe640 4 set_mon_vals no callback set 2023-10-30T20:45:56.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f0475ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:56.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f0475ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:45:56.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f0475ffb640 10 monclient: dump: 2023-10-30T20:45:56.295 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:56.295 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:56.295 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.296 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.296 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:56.296 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:56.296 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:56.296 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:56.296 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:56.297 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:56.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f048ba15640 10 client.16269.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:56.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f048ba15640 10 monclient: _renew_subs 2023-10-30T20:45:56.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f048ba15640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:45:56.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f048ba15640 1 -- 172.21.15.69:0/1079505482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0484140d40 con 0x7f0484141010 2023-10-30T20:45:56.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.273+0000 7f048ba15640 1 librados: init done 2023-10-30T20:45:56.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.274+0000 7f048ba15640 10 client.16269.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:56.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.274+0000 7f048ba15640 15 client.16269.objecter _session_command_op_assign -1 1 2023-10-30T20:45:56.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.274+0000 7f048ba15640 20 client.16269.objecter _get_session osd=-1 returning homeless 2023-10-30T20:45:56.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.274+0000 7f048ba15640 10 client.16269.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:56.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.274+0000 7f048ba15640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:45:56.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.274+0000 7f048ba15640 1 -- 172.21.15.69:0/1079505482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f04840a90e0 con 0x7f0484141010 2023-10-30T20:45:56.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.292+0000 7f0475ffb640 1 -- 172.21.15.69:0/1079505482 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f047c02ca80 con 0x7f0484141010 2023-10-30T20:45:56.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.293+0000 7f0475ffb640 1 --2- 172.21.15.69:0/1079505482 >> v2:172.21.15.69:6800/1876307353 conn(0x7f046806c7c0 0x7f046806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:56.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.293+0000 7f04897b1640 1 --2- 172.21.15.69:0/1079505482 >> v2:172.21.15.69:6800/1876307353 conn(0x7f046806c7c0 0x7f046806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:56.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.293+0000 7f04897b1640 10 monclient: get_auth_request con 0x7f046806c7c0 auth_method 0 2023-10-30T20:45:56.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.293+0000 7f0475ffb640 10 client.16269.objecter ms_handle_connect 0x7f046806c7c0 2023-10-30T20:45:56.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.293+0000 7f04897b1640 1 --2- 172.21.15.69:0/1079505482 >> v2:172.21.15.69:6800/1876307353 conn(0x7f046806c7c0 0x7f046806ec80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0478007fd0 tx=0x7f0478009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:56.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.294+0000 7f0475ffb640 1 -- 172.21.15.69:0/1079505482 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f047c04d030 con 0x7f0484141010 2023-10-30T20:45:56.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.294+0000 7f0475ffb640 10 client.16269.objecter ms_dispatch 0x7f0484000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:45:56.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.294+0000 7f0475ffb640 3 client.16269.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:45:56.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.294+0000 7f0475ffb640 3 client.16269.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:45:56.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.295+0000 7f0475ffb640 10 client.16269.objecter checking command 1 2023-10-30T20:45:56.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.295+0000 7f0475ffb640 1 --2- 172.21.15.69:0/1079505482 >> v2:172.21.15.69:6805/2300065874 conn(0x7f0468073c00 0x7f04680760a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:56.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.295+0000 7f0475ffb640 20 client.16269.objecter _get_session s=0x7f047c03e830 osd=2 3 2023-10-30T20:45:56.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.295+0000 7f0475ffb640 20 client.16269.objecter put_session s=0x7f047c03e830 osd=2 3 2023-10-30T20:45:56.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.295+0000 7f0475ffb640 15 client.16269.objecter _session_command_op_remove -1 1 2023-10-30T20:45:56.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.295+0000 7f0475ffb640 20 client.16269.objecter _get_session s=0x7f047c03e830 osd=2 3 2023-10-30T20:45:56.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.295+0000 7f0475ffb640 20 client.16269.objecter get_session s=0x7f047c03e830 osd=2 3 2023-10-30T20:45:56.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.295+0000 7f0475ffb640 15 client.16269.objecter _session_command_op_assign 2 1 2023-10-30T20:45:56.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.295+0000 7f0475ffb640 20 client.16269.objecter put_session s=0x7f047c03e830 osd=2 4 2023-10-30T20:45:56.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.295+0000 7f0475ffb640 10 client.16269.objecter _send_command 1 2023-10-30T20:45:56.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.295+0000 7f0475ffb640 1 -- 172.21.15.69:0/1079505482 --> v2:172.21.15.69:6805/2300065874 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f047c009510 con 0x7f0468073c00 2023-10-30T20:45:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.295+0000 7f0475ffb640 20 client.16269.objecter dump_active .. 0 homeless 2023-10-30T20:45:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.295+0000 7f0488fb0640 1 --2- 172.21.15.69:0/1079505482 >> v2:172.21.15.69:6805/2300065874 conn(0x7f0468073c00 0x7f04680760a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.295+0000 7f0488fb0640 10 monclient: get_auth_request con 0x7f0468073c00 auth_method 0 2023-10-30T20:45:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.298+0000 7f0475ffb640 10 client.16269.objecter ms_handle_connect 0x7f0468073c00 2023-10-30T20:45:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.298+0000 7f0488fb0640 1 --2- 172.21.15.69:0/1079505482 >> v2:172.21.15.69:6805/2300065874 conn(0x7f0468073c00 0x7f04680760a0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:56.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.300+0000 7f0475ffb640 1 -- 172.21.15.69:0/1079505482 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=904) v2 ==== 24+0+0 (secure 0 0 0) 0x7f047c009b30 con 0x7f0484141010 2023-10-30T20:45:56.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.300+0000 7f0475ffb640 10 monclient: handle_get_version_reply finishing 1 version 904 2023-10-30T20:45:56.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.322+0000 7f0475ffb640 1 -- 172.21.15.69:0/1079505482 <== osd.2 v2:172.21.15.69:6805/2300065874 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f047c009510 con 0x7f0468073c00 2023-10-30T20:45:56.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.322+0000 7f0475ffb640 10 client.16269.objecter ms_dispatch 0x7f0484000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:45:56.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.322+0000 7f0475ffb640 10 client.16269.objecter _finish_command 1 = system:0 2023-10-30T20:45:56.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.322+0000 7f0475ffb640 20 client.16269.objecter put_session s=0x7f047c03e830 osd=2 4 2023-10-30T20:45:56.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.322+0000 7f0475ffb640 15 client.16269.objecter _session_command_op_remove 2 1 2023-10-30T20:45:56.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.348+0000 7f048ba15640 10 client.16269.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T20:45:56.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.348+0000 7f048ba15640 15 client.16269.objecter _session_command_op_assign -1 2 2023-10-30T20:45:56.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.348+0000 7f048ba15640 20 client.16269.objecter _get_session s=0x7f047c03e830 osd=2 3 2023-10-30T20:45:56.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.348+0000 7f048ba15640 20 client.16269.objecter put_session s=0x7f047c03e830 osd=2 3 2023-10-30T20:45:56.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.348+0000 7f048ba15640 20 client.16269.objecter _get_session s=0x7f047c03e830 osd=2 3 2023-10-30T20:45:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.348+0000 7f048ba15640 15 client.16269.objecter _session_command_op_remove -1 2 2023-10-30T20:45:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.348+0000 7f048ba15640 20 client.16269.objecter get_session s=0x7f047c03e830 osd=2 3 2023-10-30T20:45:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.348+0000 7f048ba15640 15 client.16269.objecter _session_command_op_assign 2 2 2023-10-30T20:45:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.348+0000 7f048ba15640 20 client.16269.objecter put_session s=0x7f047c03e830 osd=2 4 2023-10-30T20:45:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.348+0000 7f048ba15640 10 client.16269.objecter _send_command 2 2023-10-30T20:45:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.348+0000 7f048ba15640 1 -- 172.21.15.69:0/1079505482 --> v2:172.21.15.69:6805/2300065874 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f04840aa720 con 0x7f0468073c00 2023-10-30T20:45:56.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.369+0000 7f0475ffb640 1 -- 172.21.15.69:0/1079505482 <== osd.2 v2:172.21.15.69:6805/2300065874 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f0480009bd0 con 0x7f0468073c00 2023-10-30T20:45:56.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.369+0000 7f0475ffb640 10 client.16269.objecter ms_dispatch 0x7f0484000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:45:56.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.369+0000 7f0475ffb640 10 client.16269.objecter _finish_command 2 = system:0 2023-10-30T20:45:56.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.369+0000 7f0475ffb640 20 client.16269.objecter put_session s=0x7f047c03e830 osd=2 4 2023-10-30T20:45:56.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.369+0000 7f0475ffb640 15 client.16269.objecter _session_command_op_remove 2 2 2023-10-30T20:45:56.372 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-10-30T20:45:56.372 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T20:45:56.372 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-10-30T20:45:56.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 10 librados: watch_flush enter 2023-10-30T20:45:56.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 10 librados: watch_flush exit 2023-10-30T20:45:56.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 10 client.16269.objecter close_session for osd.2 2023-10-30T20:45:56.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 1 -- 172.21.15.69:0/1079505482 >> v2:172.21.15.69:6805/2300065874 conn(0x7f0468073c00 msgr2=0x7f04680760a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 1 --2- 172.21.15.69:0/1079505482 >> v2:172.21.15.69:6805/2300065874 conn(0x7f0468073c00 0x7f04680760a0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 20 client.16269.objecter put_session s=0x7f047c03e830 osd=2 1 2023-10-30T20:45:56.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 20 client.16269.objecter shutdown clearing up homeless session... 2023-10-30T20:45:56.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 10 client.16269.objecter successfully canceled tick 2023-10-30T20:45:56.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 1 -- 172.21.15.69:0/1079505482 >> v2:172.21.15.69:6800/1876307353 conn(0x7f046806c7c0 msgr2=0x7f046806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 1 --2- 172.21.15.69:0/1079505482 >> v2:172.21.15.69:6800/1876307353 conn(0x7f046806c7c0 0x7f046806ec80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0478007fd0 tx=0x7f0478009040 comp rx=0 tx=0).stop 2023-10-30T20:45:56.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 10 monclient: shutdown 2023-10-30T20:45:56.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:56.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 1 -- 172.21.15.69:0/1079505482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0484141010 msgr2=0x7f04841d7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 1 --2- 172.21.15.69:0/1079505482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0484141010 0x7f04841d7050 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 crypto rx=0x7f04840aab80 tx=0x7f047c004a40 comp rx=0 tx=0).stop 2023-10-30T20:45:56.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f0489fb2640 1 -- 172.21.15.69:0/1079505482 reap_dead start 2023-10-30T20:45:56.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 1 -- 172.21.15.69:0/1079505482 shutdown_connections 2023-10-30T20:45:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 1 -- 172.21.15.69:0/1079505482 >> 172.21.15.69:0/1079505482 conn(0x7f0484065ed0 msgr2=0x7f04840b70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 1 -- 172.21.15.69:0/1079505482 shutdown_connections 2023-10-30T20:45:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 1 -- 172.21.15.69:0/1079505482 wait complete. 2023-10-30T20:45:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.370+0000 7f048ba15640 1 librados: shutdown 2023-10-30T20:45:56.382 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-30T20:45:56.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.480+0000 7fd99a533640 10 monclient: get_monmap_and_config 2023-10-30T20:45:56.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.480+0000 7fd99a533640 10 monclient: build_initial_monmap 2023-10-30T20:45:56.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.480+0000 7fd99a533640 10 monclient: monmap: 2023-10-30T20:45:56.482 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:56.482 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:56.482 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:56.481039+0000 2023-10-30T20:45:56.482 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:56.481039+0000 2023-10-30T20:45:56.482 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:56.483 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:56.483 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:56.483 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:56.483 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:56.483 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:56.484 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:56.484 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:56.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.480+0000 7fd99a533640 1 Processor -- start 2023-10-30T20:45:56.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.480+0000 7fd99a533640 1 -- start start 2023-10-30T20:45:56.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.480+0000 7fd99a533640 10 monclient: init 2023-10-30T20:45:56.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.481+0000 7fd99a533640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:56.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.481+0000 7fd99a533640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-10-30T20:45:56.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.481+0000 7fd99a533640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9940af190 0x7fd994070d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:56.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.481+0000 7fd99a533640 10 monclient(hunting): picked mon.noname-b con 0x7fd9940af190 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:56.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.481+0000 7fd99a533640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd99406f5d0 0x7fd994071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:56.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.481+0000 7fd99a533640 10 monclient(hunting): picked mon.noname-c con 0x7fd99406f5d0 addr v2:172.21.15.183:3300/0 2023-10-30T20:45:56.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.481+0000 7fd99a533640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd9940717b0 addr v1:172.21.15.183:6789/0 2023-10-30T20:45:56.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.481+0000 7fd99a533640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd994070630 con 0x7fd9940717b0 2023-10-30T20:45:56.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.481+0000 7fd99a533640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:56.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.481+0000 7fd99a533640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd994067b90 con 0x7fd99406f5d0 2023-10-30T20:45:56.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.481+0000 7fd99a533640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:56.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.481+0000 7fd99a533640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9940af570 con 0x7fd9940af190 2023-10-30T20:45:56.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.481+0000 7fd99a533640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:56.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.481+0000 7fd99a533640 10 monclient(hunting): authenticate will time out at 2879.770396s 2023-10-30T20:45:56.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.482+0000 7fd993fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9940af190 0x7fd994070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:56.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.482+0000 7fd9937fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd99406f5d0 0x7fd994071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:56.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.482+0000 7fd993fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9940af190 0x7fd994070d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41784/0 (socket says 172.21.15.69:41784) 2023-10-30T20:45:56.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.482+0000 7fd993fff640 1 -- 172.21.15.69:0/1783749356 learned_addr learned my addr 172.21.15.69:0/1783749356 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:56.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.482+0000 7fd993fff640 10 monclient(hunting): get_auth_request con 0x7fd9940af190 auth_method 0 2023-10-30T20:45:56.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.482+0000 7fd993fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:56.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.482+0000 7fd993fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:56.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.482+0000 7fd993fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:56.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.482+0000 7fd9937fe640 10 monclient(hunting): get_auth_request con 0x7fd99406f5d0 auth_method 0 2023-10-30T20:45:56.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.482+0000 7fd9937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:56.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.482+0000 7fd9937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:56.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.482+0000 7fd9937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:56.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.483+0000 7fd993fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:56.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.483+0000 7fd993fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:56.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.483+0000 7fd993fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:56.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.483+0000 7fd9937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:56.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.483+0000 7fd9937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:56.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.483+0000 7fd9937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:56.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.484+0000 7fd993fff640 10 monclient(hunting): handle_auth_done global_id 18308 payload 274 2023-10-30T20:45:56.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.484+0000 7fd993fff640 1 -- 172.21.15.69:0/1783749356 >> v2:172.21.15.183:3300/0 conn(0x7fd99406f5d0 msgr2=0x7fd994071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:56.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.484+0000 7fd993fff640 1 --2- 172.21.15.69:0/1783749356 >> v2:172.21.15.183:3300/0 conn(0x7fd99406f5d0 0x7fd994071270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.484+0000 7fd993fff640 1 -- 172.21.15.69:0/1783749356 >> v1:172.21.15.183:6789/0 conn(0x7fd9940717b0 legacy=0x7fd99406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.484+0000 7fd993fff640 10 monclient: _finish_hunting 0 2023-10-30T20:45:56.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.484+0000 7fd993fff640 1 monclient: found mon.noname-b 2023-10-30T20:45:56.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.484+0000 7fd993fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:56.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.484+0000 7fd993fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:45:56.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.484+0000 7fd993fff640 1 -- 172.21.15.69:0/1783749356 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd994070910 con 0x7fd9940af190 2023-10-30T20:45:56.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.486+0000 7fd993fff640 1 --2- 172.21.15.69:0/1783749356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9940af190 0x7fd994070d30 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 crypto rx=0x7fd98400bb10 tx=0x7fd98402e150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6852b8787c53402 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:56.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd992ffd640 1 -- 172.21.15.69:0/1783749356 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd984036440 con 0x7fd9940af190 2023-10-30T20:45:56.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd992ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:56.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd992ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:56.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd992ffd640 10 monclient: dump: 2023-10-30T20:45:56.498 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:56.498 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:56.498 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.498 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.499 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:56.499 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:56.499 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:56.499 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:56.499 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:56.499 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:56.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd992ffd640 10 monclient: _finish_auth 0 2023-10-30T20:45:56.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd992ffd640 10 monclient: _check_auth_tickets 2023-10-30T20:45:56.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd992ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:56.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd992ffd640 1 -- 172.21.15.69:0/1783749356 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9840365b0 con 0x7fd9940af190 2023-10-30T20:45:56.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd992ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:56.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd992ffd640 1 -- 172.21.15.69:0/1783749356 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd984036950 con 0x7fd9940af190 2023-10-30T20:45:56.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd992ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:56.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd992ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:45:56.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd992ffd640 10 monclient: dump: 2023-10-30T20:45:56.501 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:56.501 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:56.502 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.502 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.502 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:56.502 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:56.502 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:56.502 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:56.503 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:56.503 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:56.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd99a533640 5 monclient: authenticate success, global_id 18308 2023-10-30T20:45:56.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd99a533640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:56.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.490+0000 7fd99a533640 4 set_mon_vals no callback set 2023-10-30T20:45:56.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.491+0000 7fd99a533640 10 monclient: shutdown 2023-10-30T20:45:56.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.491+0000 7fd99a533640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:56.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.491+0000 7fd99a533640 1 -- 172.21.15.69:0/1783749356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9940af190 msgr2=0x7fd994070d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.491+0000 7fd99a533640 1 --2- 172.21.15.69:0/1783749356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9940af190 0x7fd994070d30 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 crypto rx=0x7fd98400bb10 tx=0x7fd98402e150 comp rx=0 tx=0).stop 2023-10-30T20:45:56.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.491+0000 7fd99a533640 1 -- 172.21.15.69:0/1783749356 shutdown_connections 2023-10-30T20:45:56.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.491+0000 7fd99a533640 1 --2- 172.21.15.69:0/1783749356 >> v2:172.21.15.183:3300/0 conn(0x7fd99406f5d0 0x7fd994071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.491+0000 7fd99a533640 1 --2- 172.21.15.69:0/1783749356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9940af190 0x7fd994070d30 unknown :-1 s=CLOSED pgs=3210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.491+0000 7fd99a533640 1 -- 172.21.15.69:0/1783749356 >> 172.21.15.69:0/1783749356 conn(0x7fd994069d50 msgr2=0x7fd99406a130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:56.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.491+0000 7fd99a533640 1 -- 172.21.15.69:0/1783749356 shutdown_connections 2023-10-30T20:45:56.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.491+0000 7fd99a533640 1 -- 172.21.15.69:0/1783749356 wait complete. 2023-10-30T20:45:56.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.491+0000 7fd99a533640 10 monclient: build_initial_monmap 2023-10-30T20:45:56.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.491+0000 7fd99a533640 10 monclient: monmap: 2023-10-30T20:45:56.506 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:56.506 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:56.506 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:56.506 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:56.507 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:56.507 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:56.507 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:56.507 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:56.507 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:56.507 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:56.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.492+0000 7fd99a533640 1 librados: starting msgr at 2023-10-30T20:45:56.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.492+0000 7fd99a533640 1 librados: starting objecter 2023-10-30T20:45:56.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.492+0000 7fd99a533640 1 Processor -- start 2023-10-30T20:45:56.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.492+0000 7fd99a533640 1 -- start start 2023-10-30T20:45:56.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.492+0000 7fd99a533640 1 librados: setting wanted keys 2023-10-30T20:45:56.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.492+0000 7fd99a533640 1 librados: calling monclient init 2023-10-30T20:45:56.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.492+0000 7fd99a533640 10 monclient: init 2023-10-30T20:45:56.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.492+0000 7fd99a533640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:56.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.492+0000 7fd99a533640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T20:45:56.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.492+0000 7fd99a533640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9940717b0 0x7fd9940aca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:56.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd99a533640 10 monclient(hunting): picked mon.noname-c con 0x7fd9940717b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:56.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd99a533640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9940af190 0x7fd9941d9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:56.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd99a533640 10 monclient(hunting): picked mon.noname-a con 0x7fd9940af190 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:56.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd99a533640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd994136ff0 0x7fd9941da1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:56.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd99a533640 10 monclient(hunting): picked mon.noname-b con 0x7fd994136ff0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:56.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd99a533640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:56.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd99a533640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99406f010 con 0x7fd9940af190 2023-10-30T20:45:56.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd99a533640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:56.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd99a533640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9941417e0 con 0x7fd994136ff0 2023-10-30T20:45:56.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd99a533640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:56.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd99a533640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd994136420 con 0x7fd9940717b0 2023-10-30T20:45:56.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd99a533640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:56.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd99a533640 10 monclient(hunting): authenticate will time out at 2879.781886s 2023-10-30T20:45:56.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd993fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9940717b0 0x7fd9940aca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:56.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd9937fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9940af190 0x7fd9941d9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:56.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd998ad0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd994136ff0 0x7fd9941da1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:56.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd993fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9940717b0 0x7fd9940aca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41796/0 (socket says 172.21.15.69:41796) 2023-10-30T20:45:56.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd993fff640 1 -- 172.21.15.69:0/2362141182 learned_addr learned my addr 172.21.15.69:0/2362141182 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:56.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd993fff640 10 monclient(hunting): get_auth_request con 0x7fd9940717b0 auth_method 0 2023-10-30T20:45:56.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd993fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:56.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd993fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:56.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd993fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:56.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.493+0000 7fd998ad0640 10 monclient(hunting): get_auth_request con 0x7fd994136ff0 auth_method 0 2023-10-30T20:45:56.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd998ad0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:56.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd998ad0640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:56.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd998ad0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:56.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd9937fe640 10 monclient(hunting): get_auth_request con 0x7fd9940af190 auth_method 0 2023-10-30T20:45:56.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd9937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:56.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd9937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:56.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd9937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:56.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd993fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:56.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd993fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:56.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd993fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:56.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd998ad0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:56.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd998ad0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:56.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd998ad0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:56.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd9937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:56.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd9937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:56.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.494+0000 7fd9937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:56.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.497+0000 7fd998ad0640 10 monclient(hunting): handle_auth_done global_id 21247 payload 824 2023-10-30T20:45:56.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.497+0000 7fd998ad0640 1 -- 172.21.15.69:0/2362141182 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9940717b0 msgr2=0x7fd9940aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.497+0000 7fd998ad0640 1 --2- 172.21.15.69:0/2362141182 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9940717b0 0x7fd9940aca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.497+0000 7fd998ad0640 1 -- 172.21.15.69:0/2362141182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9940af190 msgr2=0x7fd9941d9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.497+0000 7fd998ad0640 1 --2- 172.21.15.69:0/2362141182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9940af190 0x7fd9941d9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.497+0000 7fd998ad0640 10 monclient: _finish_hunting 0 2023-10-30T20:45:56.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.497+0000 7fd998ad0640 1 monclient: found mon.noname-b 2023-10-30T20:45:56.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.497+0000 7fd998ad0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:56.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.497+0000 7fd998ad0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:45:56.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.497+0000 7fd998ad0640 1 -- 172.21.15.69:0/2362141182 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd994141ef0 con 0x7fd994136ff0 2023-10-30T20:45:56.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.499+0000 7fd998ad0640 1 --2- 172.21.15.69:0/2362141182 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd994136ff0 0x7fd9941da1f0 secure :-1 s=READY pgs=5050 cs=0 l=1 rev1=1 crypto rx=0x7fd994071c40 tx=0x7fd98c00cb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:56.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.499+0000 7fd990ff9640 10 client.?.objecter ms_handle_connect 0x7fd994136ff0 2023-10-30T20:45:56.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.499+0000 7fd990ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:56.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd990ff9640 1 -- 172.21.15.69:0/2362141182 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c003b70 con 0x7fd994136ff0 2023-10-30T20:45:56.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd990ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:56.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd990ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:56.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd990ff9640 10 monclient: dump: 2023-10-30T20:45:56.521 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:56.521 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:56.521 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.522 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.522 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:56.522 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:56.522 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:56.522 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:56.523 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:56.523 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:56.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd990ff9640 10 monclient: _finish_auth 0 2023-10-30T20:45:56.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd990ff9640 10 monclient: _check_auth_tickets 2023-10-30T20:45:56.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd990ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd990ff9640 1 -- 172.21.15.69:0/2362141182 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd98c00e040 con 0x7fd994136ff0 2023-10-30T20:45:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd990ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd990ff9640 1 -- 172.21.15.69:0/2362141182 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c0050d0 con 0x7fd994136ff0 2023-10-30T20:45:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd99a533640 5 monclient: authenticate success, global_id 21247 2023-10-30T20:45:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd99a533640 10 monclient: _renew_subs 2023-10-30T20:45:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd99a533640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd99a533640 1 -- 172.21.15.69:0/2362141182 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd994142150 con 0x7fd994136ff0 2023-10-30T20:45:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd9927fc640 4 set_mon_vals no callback set 2023-10-30T20:45:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd990ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd990ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd990ff9640 10 monclient: dump: 2023-10-30T20:45:56.526 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:56.526 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:56.526 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.526 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:56.526 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:56.527 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd99a533640 10 client.21247.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:56.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd99a533640 10 monclient: _renew_subs 2023-10-30T20:45:56.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd99a533640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:56.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd99a533640 1 -- 172.21.15.69:0/2362141182 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd99413e4e0 con 0x7fd994136ff0 2023-10-30T20:45:56.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.503+0000 7fd99a533640 1 librados: init done 2023-10-30T20:45:56.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.504+0000 7fd99a533640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:56.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.504+0000 7fd99a533640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:56.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.504+0000 7fd99a533640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:56.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.504+0000 7fd99a533640 1 -- 172.21.15.69:0/2362141182 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99404e9b0 con 0x7fd994136ff0 2023-10-30T20:45:56.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.523+0000 7fd990ff9640 1 -- 172.21.15.69:0/2362141182 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fd98c005240 con 0x7fd994136ff0 2023-10-30T20:45:56.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.523+0000 7fd990ff9640 1 --2- 172.21.15.69:0/2362141182 >> v2:172.21.15.69:6800/1876307353 conn(0x7fd97406c810 0x7fd97406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:56.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.523+0000 7fd993fff640 1 --2- 172.21.15.69:0/2362141182 >> v2:172.21.15.69:6800/1876307353 conn(0x7fd97406c810 0x7fd97406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.523+0000 7fd993fff640 10 monclient: get_auth_request con 0x7fd97406c810 auth_method 0 2023-10-30T20:45:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.524+0000 7fd990ff9640 10 client.21247.objecter ms_handle_connect 0x7fd97406c810 2023-10-30T20:45:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.524+0000 7fd993fff640 1 --2- 172.21.15.69:0/2362141182 >> v2:172.21.15.69:6800/1876307353 conn(0x7fd97406c810 0x7fd97406ecd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd984039040 tx=0x7fd984008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.525+0000 7fd990ff9640 1 -- 172.21.15.69:0/2362141182 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7fd98c0584e0 con 0x7fd994136ff0 2023-10-30T20:45:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.525+0000 7fd990ff9640 10 client.21247.objecter ms_dispatch 0x7fd994000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:45:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.525+0000 7fd990ff9640 3 client.21247.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:45:56.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.525+0000 7fd990ff9640 3 client.21247.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:45:56.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.525+0000 7fd990ff9640 20 client.21247.objecter dump_active .. 0 homeless 2023-10-30T20:45:56.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.758+0000 7fd990ff9640 1 -- 172.21.15.69:0/2362141182 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd98c05c580 con 0x7fd994136ff0 2023-10-30T20:45:56.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.758+0000 7fd990ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:56.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.758+0000 7fd990ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:45:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.929+0000 7fd99a533640 1 -- 172.21.15.69:0/2362141182 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9940ae690 con 0x7fd97406c810 2023-10-30T20:45:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.930+0000 7fd990ff9640 1 -- 172.21.15.69:0/2362141182 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd984036650 con 0x7fd97406c810 2023-10-30T20:45:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 2 to deep-scrub 2023-10-30T20:45:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 10 librados: watch_flush enter 2023-10-30T20:45:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 10 librados: watch_flush exit 2023-10-30T20:45:56.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 20 client.21247.objecter shutdown clearing up homeless session... 2023-10-30T20:45:56.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 10 client.21247.objecter successfully canceled tick 2023-10-30T20:45:56.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 1 -- 172.21.15.69:0/2362141182 >> v2:172.21.15.69:6800/1876307353 conn(0x7fd97406c810 msgr2=0x7fd97406ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 1 --2- 172.21.15.69:0/2362141182 >> v2:172.21.15.69:6800/1876307353 conn(0x7fd97406c810 0x7fd97406ecd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd984039040 tx=0x7fd984008040 comp rx=0 tx=0).stop 2023-10-30T20:45:56.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 10 monclient: shutdown 2023-10-30T20:45:56.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:56.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 1 -- 172.21.15.69:0/2362141182 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd994136ff0 msgr2=0x7fd9941da1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:56.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 1 --2- 172.21.15.69:0/2362141182 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd994136ff0 0x7fd9941da1f0 secure :-1 s=READY pgs=5050 cs=0 l=1 rev1=1 crypto rx=0x7fd994071c40 tx=0x7fd98c00cb20 comp rx=0 tx=0).stop 2023-10-30T20:45:56.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 1 -- 172.21.15.69:0/2362141182 shutdown_connections 2023-10-30T20:45:56.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 1 --2- 172.21.15.69:0/2362141182 >> v2:172.21.15.69:6800/1876307353 conn(0x7fd97406c810 0x7fd97406ecd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 1 --2- 172.21.15.69:0/2362141182 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd994136ff0 0x7fd9941da1f0 unknown :-1 s=CLOSED pgs=5050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 1 --2- 172.21.15.69:0/2362141182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9940af190 0x7fd9941d9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 1 --2- 172.21.15.69:0/2362141182 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9940717b0 0x7fd9940aca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 1 -- 172.21.15.69:0/2362141182 >> 172.21.15.69:0/2362141182 conn(0x7fd994069d50 msgr2=0x7fd99406d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 1 -- 172.21.15.69:0/2362141182 shutdown_connections 2023-10-30T20:45:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.932+0000 7fd99a533640 1 -- 172.21.15.69:0/2362141182 wait complete. 2023-10-30T20:45:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:56.933+0000 7fd99a533640 1 librados: shutdown 2023-10-30T20:45:56.945 INFO:tasks.ceph:Scrubbing osd.3 2023-10-30T20:45:56.945 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-30T20:45:57.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.046+0000 7f6d15c94640 10 monclient: get_monmap_and_config 2023-10-30T20:45:57.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.046+0000 7f6d15c94640 10 monclient: build_initial_monmap 2023-10-30T20:45:57.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.046+0000 7f6d15c94640 10 monclient: monmap: 2023-10-30T20:45:57.048 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:57.048 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.049 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:57.047822+0000 2023-10-30T20:45:57.049 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:57.047822+0000 2023-10-30T20:45:57.049 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:57.049 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.049 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:57.049 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:57.050 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:57.050 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:57.050 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:57.050 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.047+0000 7f6d15c94640 1 Processor -- start 2023-10-30T20:45:57.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.047+0000 7f6d15c94640 1 -- start start 2023-10-30T20:45:57.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.047+0000 7f6d15c94640 10 monclient: init 2023-10-30T20:45:57.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d15c94640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:57.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d15c94640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-10-30T20:45:57.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d15c94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d100ad960 addr v1:172.21.15.183:6789/0 2023-10-30T20:45:57.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d15c94640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6d100add40 0x7f6d1013e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d15c94640 10 monclient(hunting): picked mon.noname-c con 0x7f6d100add40 addr v2:172.21.15.183:3300/0 2023-10-30T20:45:57.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d15c94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d100ad230 addr v1:172.21.15.69:6789/0 2023-10-30T20:45:57.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d15c94640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d10071c60 con 0x7f6d100ad230 2023-10-30T20:45:57.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d15c94640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d10070d30 con 0x7f6d100ad960 2023-10-30T20:45:57.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d15c94640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:57.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d15c94640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d10074040 con 0x7f6d100add40 2023-10-30T20:45:57.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d15c94640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:57.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d15c94640 10 monclient(hunting): authenticate will time out at 2880.337072s 2023-10-30T20:45:57.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d0ffff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6d100ad230 0x7f6d1006ed20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:38940/0 (socket says 172.21.15.69:38940) 2023-10-30T20:45:57.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d0ffff640 1 -- 172.21.15.69:0/2286219884 learned_addr learned my addr 172.21.15.69:0/2286219884 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:45:57.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.048+0000 7f6d06ffd640 1 --2- 172.21.15.69:0/2286219884 >> v2:172.21.15.183:3300/0 conn(0x7f6d100add40 0x7f6d1013e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.049+0000 7f6d06ffd640 10 monclient(hunting): get_auth_request con 0x7f6d100add40 auth_method 0 2023-10-30T20:45:57.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.049+0000 7f6d06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:57.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.049+0000 7f6d06ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.049+0000 7f6d06ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.050+0000 7f6d06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:57.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.050+0000 7f6d06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:57.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.050+0000 7f6d06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:57.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.051+0000 7f6d0effd640 1 -- 172.21.15.69:0/2286219884 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606890611 0 0) 0x7f6d08001490 con 0x7f6d100ad230 2023-10-30T20:45:57.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.051+0000 7f6d0effd640 10 monclient(hunting): _negotiate 2023-10-30T20:45:57.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.051+0000 7f6d0effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.051+0000 7f6d0effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.051+0000 7f6d0effd640 10 monclient(hunting): my global_id is 16275 2023-10-30T20:45:57.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.051+0000 7f6d0effd640 1 -- 172.21.15.69:0/2286219884 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cfc005730 con 0x7f6d100ad230 2023-10-30T20:45:57.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.052+0000 7f6d06ffd640 10 monclient(hunting): handle_auth_done global_id 21250 payload 274 2023-10-30T20:45:57.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.052+0000 7f6d06ffd640 1 -- 172.21.15.69:0/2286219884 >> v1:172.21.15.183:6789/0 conn(0x7f6d100ad960 legacy=0x7f6d100ac980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.052+0000 7f6d06ffd640 1 -- 172.21.15.69:0/2286219884 >> v1:172.21.15.69:6789/0 conn(0x7f6d100ad230 legacy=0x7f6d1006ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.052+0000 7f6d06ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:45:57.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.052+0000 7f6d06ffd640 1 monclient: found mon.noname-c 2023-10-30T20:45:57.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.052+0000 7f6d06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:57.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.052+0000 7f6d06ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:45:57.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.052+0000 7f6d06ffd640 1 -- 172.21.15.69:0/2286219884 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1006f430 con 0x7f6d100add40 2023-10-30T20:45:57.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.057+0000 7f6d06ffd640 1 --2- 172.21.15.69:0/2286219884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d100add40 0x7f6d1013e620 secure :-1 s=READY pgs=5052 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80097a0 tx=0x7f6cf802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=363a9c48aaeb4307 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:57.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.058+0000 7f6d0effd640 1 -- 172.21.15.69:0/2286219884 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8031830 con 0x7f6d100add40 2023-10-30T20:45:57.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.058+0000 7f6d0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:57.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.058+0000 7f6d0effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:45:57.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.058+0000 7f6d0effd640 10 monclient: dump: 2023-10-30T20:45:57.061 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:57.061 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.061 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.061 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.062 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:57.062 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.062 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:57.062 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:57.062 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:57.062 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.059+0000 7f6d0effd640 10 monclient: _finish_auth 0 2023-10-30T20:45:57.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.059+0000 7f6d0effd640 10 monclient: _check_auth_tickets 2023-10-30T20:45:57.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.059+0000 7f6d0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:57.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.059+0000 7f6d15c94640 5 monclient: authenticate success, global_id 21250 2023-10-30T20:45:57.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.059+0000 7f6d15c94640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:45:57.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.060+0000 7f6d0effd640 1 -- 172.21.15.69:0/2286219884 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf802d040 con 0x7f6d100add40 2023-10-30T20:45:57.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.060+0000 7f6d0effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:57.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.060+0000 7f6d0effd640 1 -- 172.21.15.69:0/2286219884 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8031ba0 con 0x7f6d100add40 2023-10-30T20:45:57.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.060+0000 7f6d0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:57.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.060+0000 7f6d0effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:57.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.060+0000 7f6d0effd640 10 monclient: dump: 2023-10-30T20:45:57.067 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:57.067 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.068 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.068 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.068 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:57.068 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.068 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:57.069 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.060+0000 7f6d15c94640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.060+0000 7f6d15c94640 4 set_mon_vals no callback set 2023-10-30T20:45:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.061+0000 7f6d15c94640 10 monclient: shutdown 2023-10-30T20:45:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.061+0000 7f6d15c94640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.061+0000 7f6d15c94640 1 -- 172.21.15.69:0/2286219884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d100add40 msgr2=0x7f6d1013e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.061+0000 7f6d15c94640 1 --2- 172.21.15.69:0/2286219884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d100add40 0x7f6d1013e620 secure :-1 s=READY pgs=5052 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80097a0 tx=0x7f6cf802b8d0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.061+0000 7f6d15c94640 1 -- 172.21.15.69:0/2286219884 shutdown_connections 2023-10-30T20:45:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.061+0000 7f6d15c94640 1 --2- 172.21.15.69:0/2286219884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d100add40 0x7f6d1013e620 unknown :-1 s=CLOSED pgs=5052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.061+0000 7f6d15c94640 1 -- 172.21.15.69:0/2286219884 >> 172.21.15.69:0/2286219884 conn(0x7f6d100afe50 msgr2=0x7f6d100b0260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:57.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.061+0000 7f6d15c94640 1 -- 172.21.15.69:0/2286219884 shutdown_connections 2023-10-30T20:45:57.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.061+0000 7f6d15c94640 1 -- 172.21.15.69:0/2286219884 wait complete. 2023-10-30T20:45:57.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.062+0000 7f6d15c94640 10 monclient: build_initial_monmap 2023-10-30T20:45:57.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.062+0000 7f6d15c94640 10 monclient: monmap: 2023-10-30T20:45:57.071 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:57.073 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:57.073 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:57.073 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.062+0000 7f6d15c94640 1 librados: starting msgr at 2023-10-30T20:45:57.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.062+0000 7f6d15c94640 1 librados: starting objecter 2023-10-30T20:45:57.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.062+0000 7f6d15c94640 1 Processor -- start 2023-10-30T20:45:57.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.062+0000 7f6d15c94640 1 -- start start 2023-10-30T20:45:57.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.062+0000 7f6d15c94640 1 librados: setting wanted keys 2023-10-30T20:45:57.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.062+0000 7f6d15c94640 1 librados: calling monclient init 2023-10-30T20:45:57.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.062+0000 7f6d15c94640 10 monclient: init 2023-10-30T20:45:57.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:45:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d100ad960 0x7f6d101da360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 10 monclient(hunting): picked mon.noname-a con 0x7f6d100ad960 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d10138300 0x7f6d101da8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 10 monclient(hunting): picked mon.noname-b con 0x7f6d10138300 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1011f7d0 0x7f6d1011fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 10 monclient(hunting): picked mon.noname-c con 0x7f6d1011f7d0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1006e4d0 con 0x7f6d100ad960 2023-10-30T20:45:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d101dafe0 con 0x7f6d10138300 2023-10-30T20:45:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d10120190 con 0x7f6d1011f7d0 2023-10-30T20:45:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d15c94640 10 monclient(hunting): authenticate will time out at 2880.352084s 2023-10-30T20:45:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d0f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d100ad960 0x7f6d101da360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.063+0000 7f6d0ffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1011f7d0 0x7f6d1011fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d06ffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d10138300 0x7f6d101da8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d100ad960 0x7f6d101da360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50712/0 (socket says 172.21.15.69:50712) 2023-10-30T20:45:57.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0f7fe640 1 -- 172.21.15.69:0/3178355224 learned_addr learned my addr 172.21.15.69:0/3178355224 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:57.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0f7fe640 10 monclient(hunting): get_auth_request con 0x7f6d100ad960 auth_method 0 2023-10-30T20:45:57.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:57.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0ffff640 10 monclient(hunting): get_auth_request con 0x7f6d1011f7d0 auth_method 0 2023-10-30T20:45:57.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:57.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d06ffd640 10 monclient(hunting): get_auth_request con 0x7f6d10138300 auth_method 0 2023-10-30T20:45:57.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:57.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d06ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d06ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:57.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.064+0000 7f6d0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.067+0000 7f6d0f7fe640 10 monclient(hunting): handle_auth_done global_id 16278 payload 824 2023-10-30T20:45:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.067+0000 7f6d0f7fe640 1 -- 172.21.15.69:0/3178355224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1011f7d0 msgr2=0x7f6d1011fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.067+0000 7f6d0f7fe640 1 --2- 172.21.15.69:0/3178355224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1011f7d0 0x7f6d1011fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.067+0000 7f6d0f7fe640 1 -- 172.21.15.69:0/3178355224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d10138300 msgr2=0x7f6d101da8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.067+0000 7f6d0f7fe640 1 --2- 172.21.15.69:0/3178355224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d10138300 0x7f6d101da8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.067+0000 7f6d0f7fe640 10 monclient: _finish_hunting 0 2023-10-30T20:45:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.067+0000 7f6d0f7fe640 1 monclient: found mon.noname-a 2023-10-30T20:45:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.067+0000 7f6d0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.067+0000 7f6d0f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:45:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.067+0000 7f6d0f7fe640 1 -- 172.21.15.69:0/3178355224 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d10120360 con 0x7f6d100ad960 2023-10-30T20:45:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.070+0000 7f6d0f7fe640 1 --2- 172.21.15.69:0/3178355224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d100ad960 0x7f6d101da360 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000c480 tx=0x7f6d0000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.070+0000 7f6d0cff9640 10 client.?.objecter ms_handle_connect 0x7f6d100ad960 2023-10-30T20:45:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.070+0000 7f6d0cff9640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d0cff9640 1 -- 172.21.15.69:0/3178355224 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d000052d0 con 0x7f6d100ad960 2023-10-30T20:45:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d0cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d0cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:45:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d0cff9640 10 monclient: dump: 2023-10-30T20:45:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:57.087 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.087 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.087 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.087 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:57.087 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.087 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:57.088 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:57.088 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:57.088 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d0cff9640 10 monclient: _finish_auth 0 2023-10-30T20:45:57.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d0cff9640 10 monclient: _check_auth_tickets 2023-10-30T20:45:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d0cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d0cff9640 1 -- 172.21.15.69:0/3178355224 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0000e040 con 0x7f6d100ad960 2023-10-30T20:45:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d15c94640 5 monclient: authenticate success, global_id 16278 2023-10-30T20:45:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d15c94640 10 monclient: _renew_subs 2023-10-30T20:45:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d15c94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:45:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d15c94640 1 -- 172.21.15.69:0/3178355224 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d101d6bf0 con 0x7f6d100ad960 2023-10-30T20:45:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d0cff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d0cff9640 1 -- 172.21.15.69:0/3178355224 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d00012630 con 0x7f6d100ad960 2023-10-30T20:45:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d0cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d0cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:45:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d0cff9640 10 monclient: dump: 2023-10-30T20:45:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.092 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:57.092 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:57.092 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:57.092 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d0e7fc640 4 set_mon_vals no callback set 2023-10-30T20:45:57.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d15c94640 10 client.16278.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d15c94640 10 monclient: _renew_subs 2023-10-30T20:45:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d15c94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:45:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d15c94640 1 -- 172.21.15.69:0/3178355224 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d101d70e0 con 0x7f6d100ad960 2023-10-30T20:45:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.076+0000 7f6d15c94640 1 librados: init done 2023-10-30T20:45:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.077+0000 7f6d15c94640 10 client.16278.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:57.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.077+0000 7f6d15c94640 15 client.16278.objecter _session_command_op_assign -1 1 2023-10-30T20:45:57.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.077+0000 7f6d15c94640 20 client.16278.objecter _get_session osd=-1 returning homeless 2023-10-30T20:45:57.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.077+0000 7f6d15c94640 10 client.16278.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:57.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.077+0000 7f6d15c94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:45:57.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.077+0000 7f6d15c94640 1 -- 172.21.15.69:0/3178355224 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6d10071280 con 0x7f6d100ad960 2023-10-30T20:45:57.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.095+0000 7f6d0cff9640 1 -- 172.21.15.69:0/3178355224 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f6d000127a0 con 0x7f6d100ad960 2023-10-30T20:45:57.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.096+0000 7f6d0cff9640 1 --2- 172.21.15.69:0/3178355224 >> v2:172.21.15.69:6800/1876307353 conn(0x7f6cf006c810 0x7f6cf006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.096+0000 7f6d06ffd640 1 --2- 172.21.15.69:0/3178355224 >> v2:172.21.15.69:6800/1876307353 conn(0x7f6cf006c810 0x7f6cf006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.096+0000 7f6d06ffd640 10 monclient: get_auth_request con 0x7f6cf006c810 auth_method 0 2023-10-30T20:45:57.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.096+0000 7f6d06ffd640 1 --2- 172.21.15.69:0/3178355224 >> v2:172.21.15.69:6800/1876307353 conn(0x7f6cf006c810 0x7f6cf006ecd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8036000 tx=0x7f6cf803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:57.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.096+0000 7f6d0cff9640 10 client.16278.objecter ms_handle_connect 0x7f6cf006c810 2023-10-30T20:45:57.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 1 -- 172.21.15.69:0/3178355224 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f6d000581b0 con 0x7f6d100ad960 2023-10-30T20:45:57.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 10 client.16278.objecter ms_dispatch 0x7f6d10000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:45:57.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 3 client.16278.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:45:57.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 3 client.16278.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:45:57.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 10 client.16278.objecter checking command 1 2023-10-30T20:45:57.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 1 --2- 172.21.15.69:0/3178355224 >> v2:172.21.15.69:6809/1824616966 conn(0x7f6cf0073c50 0x7f6cf00760f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 20 client.16278.objecter _get_session s=0x7f6d00012630 osd=3 3 2023-10-30T20:45:57.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 20 client.16278.objecter put_session s=0x7f6d00012630 osd=3 3 2023-10-30T20:45:57.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 15 client.16278.objecter _session_command_op_remove -1 1 2023-10-30T20:45:57.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 20 client.16278.objecter _get_session s=0x7f6d00012630 osd=3 3 2023-10-30T20:45:57.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 20 client.16278.objecter get_session s=0x7f6d00012630 osd=3 3 2023-10-30T20:45:57.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 15 client.16278.objecter _session_command_op_assign 3 1 2023-10-30T20:45:57.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 20 client.16278.objecter put_session s=0x7f6d00012630 osd=3 4 2023-10-30T20:45:57.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 10 client.16278.objecter _send_command 1 2023-10-30T20:45:57.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 1 -- 172.21.15.69:0/3178355224 --> v2:172.21.15.69:6809/1824616966 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d000052d0 con 0x7f6cf0073c50 2023-10-30T20:45:57.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 20 client.16278.objecter dump_active .. 0 homeless 2023-10-30T20:45:57.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 1 -- 172.21.15.69:0/3178355224 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=904) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d000544f0 con 0x7f6d100ad960 2023-10-30T20:45:57.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.099+0000 7f6d0cff9640 10 monclient: handle_get_version_reply finishing 1 version 904 2023-10-30T20:45:57.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.100+0000 7f6d0ffff640 1 --2- 172.21.15.69:0/3178355224 >> v2:172.21.15.69:6809/1824616966 conn(0x7f6cf0073c50 0x7f6cf00760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.100+0000 7f6d0ffff640 10 monclient: get_auth_request con 0x7f6cf0073c50 auth_method 0 2023-10-30T20:45:57.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.103+0000 7f6d0ffff640 1 --2- 172.21.15.69:0/3178355224 >> v2:172.21.15.69:6809/1824616966 conn(0x7f6cf0073c50 0x7f6cf00760f0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:57.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.103+0000 7f6d0cff9640 10 client.16278.objecter ms_handle_connect 0x7f6cf0073c50 2023-10-30T20:45:57.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.125+0000 7f6d0cff9640 1 -- 172.21.15.69:0/3178355224 <== osd.3 v2:172.21.15.69:6809/1824616966 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f6d000052d0 con 0x7f6cf0073c50 2023-10-30T20:45:57.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.125+0000 7f6d0cff9640 10 client.16278.objecter ms_dispatch 0x7f6d10000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:45:57.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.125+0000 7f6d0cff9640 10 client.16278.objecter _finish_command 1 = system:0 2023-10-30T20:45:57.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.125+0000 7f6d0cff9640 20 client.16278.objecter put_session s=0x7f6d00012630 osd=3 4 2023-10-30T20:45:57.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.125+0000 7f6d0cff9640 15 client.16278.objecter _session_command_op_remove 3 1 2023-10-30T20:45:57.143 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.142+0000 7f6d15c94640 10 client.16278.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T20:45:57.143 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.142+0000 7f6d15c94640 15 client.16278.objecter _session_command_op_assign -1 2 2023-10-30T20:45:57.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.142+0000 7f6d15c94640 20 client.16278.objecter _get_session s=0x7f6d00012630 osd=3 3 2023-10-30T20:45:57.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.142+0000 7f6d15c94640 20 client.16278.objecter put_session s=0x7f6d00012630 osd=3 3 2023-10-30T20:45:57.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.142+0000 7f6d15c94640 20 client.16278.objecter _get_session s=0x7f6d00012630 osd=3 3 2023-10-30T20:45:57.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.142+0000 7f6d15c94640 15 client.16278.objecter _session_command_op_remove -1 2 2023-10-30T20:45:57.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.142+0000 7f6d15c94640 20 client.16278.objecter get_session s=0x7f6d00012630 osd=3 3 2023-10-30T20:45:57.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.142+0000 7f6d15c94640 15 client.16278.objecter _session_command_op_assign 3 2 2023-10-30T20:45:57.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.142+0000 7f6d15c94640 20 client.16278.objecter put_session s=0x7f6d00012630 osd=3 4 2023-10-30T20:45:57.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.142+0000 7f6d15c94640 10 client.16278.objecter _send_command 2 2023-10-30T20:45:57.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.142+0000 7f6d15c94640 1 -- 172.21.15.69:0/3178355224 --> v2:172.21.15.69:6809/1824616966 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6d10070fa0 con 0x7f6cf0073c50 2023-10-30T20:45:57.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.157+0000 7f6d0cff9640 1 -- 172.21.15.69:0/3178355224 <== osd.3 v2:172.21.15.69:6809/1824616966 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f6d08001f30 con 0x7f6cf0073c50 2023-10-30T20:45:57.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.157+0000 7f6d0cff9640 10 client.16278.objecter ms_dispatch 0x7f6d10000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:45:57.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.157+0000 7f6d0cff9640 10 client.16278.objecter _finish_command 2 = system:0 2023-10-30T20:45:57.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.157+0000 7f6d0cff9640 20 client.16278.objecter put_session s=0x7f6d00012630 osd=3 4 2023-10-30T20:45:57.160 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.157+0000 7f6d0cff9640 15 client.16278.objecter _session_command_op_remove 3 2 2023-10-30T20:45:57.160 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-10-30T20:45:57.160 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T20:45:57.160 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-10-30T20:45:57.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 10 librados: watch_flush enter 2023-10-30T20:45:57.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 10 librados: watch_flush exit 2023-10-30T20:45:57.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 10 client.16278.objecter close_session for osd.3 2023-10-30T20:45:57.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 1 -- 172.21.15.69:0/3178355224 >> v2:172.21.15.69:6809/1824616966 conn(0x7f6cf0073c50 msgr2=0x7f6cf00760f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 1 --2- 172.21.15.69:0/3178355224 >> v2:172.21.15.69:6809/1824616966 conn(0x7f6cf0073c50 0x7f6cf00760f0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 20 client.16278.objecter put_session s=0x7f6d00012630 osd=3 1 2023-10-30T20:45:57.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 20 client.16278.objecter shutdown clearing up homeless session... 2023-10-30T20:45:57.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 10 client.16278.objecter successfully canceled tick 2023-10-30T20:45:57.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 1 -- 172.21.15.69:0/3178355224 >> v2:172.21.15.69:6800/1876307353 conn(0x7f6cf006c810 msgr2=0x7f6cf006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 1 --2- 172.21.15.69:0/3178355224 >> v2:172.21.15.69:6800/1876307353 conn(0x7f6cf006c810 0x7f6cf006ecd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8036000 tx=0x7f6cf803d040 comp rx=0 tx=0).stop 2023-10-30T20:45:57.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 10 monclient: shutdown 2023-10-30T20:45:57.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:57.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 1 -- 172.21.15.69:0/3178355224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d100ad960 msgr2=0x7f6d101da360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 1 --2- 172.21.15.69:0/3178355224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d100ad960 0x7f6d101da360 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000c480 tx=0x7f6d0000c940 comp rx=0 tx=0).stop 2023-10-30T20:45:57.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d0ffff640 1 -- 172.21.15.69:0/3178355224 reap_dead start 2023-10-30T20:45:57.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 1 -- 172.21.15.69:0/3178355224 shutdown_connections 2023-10-30T20:45:57.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 1 -- 172.21.15.69:0/3178355224 >> 172.21.15.69:0/3178355224 conn(0x7f6d100afe50 msgr2=0x7f6d100737b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:57.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 1 -- 172.21.15.69:0/3178355224 shutdown_connections 2023-10-30T20:45:57.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 1 -- 172.21.15.69:0/3178355224 wait complete. 2023-10-30T20:45:57.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.158+0000 7f6d15c94640 1 librados: shutdown 2023-10-30T20:45:57.170 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-30T20:45:57.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.272+0000 7f65a9a7c640 10 monclient: get_monmap_and_config 2023-10-30T20:45:57.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.272+0000 7f65a9a7c640 10 monclient: build_initial_monmap 2023-10-30T20:45:57.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.272+0000 7f65a9a7c640 10 monclient: monmap: 2023-10-30T20:45:57.273 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:57.274 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.274 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:57.273016+0000 2023-10-30T20:45:57.274 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:57.273016+0000 2023-10-30T20:45:57.274 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:57.274 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.275 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:57.275 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:57.275 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:57.275 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:57.275 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:57.275 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.272+0000 7f65a9a7c640 1 Processor -- start 2023-10-30T20:45:57.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.272+0000 7f65a9a7c640 1 -- start start 2023-10-30T20:45:57.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.272+0000 7f65a9a7c640 10 monclient: init 2023-10-30T20:45:57.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.273+0000 7f65a9a7c640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:57.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.273+0000 7f65a9a7c640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-10-30T20:45:57.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.273+0000 7f65a9a7c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f65a40a8e40 0x7f65a413efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.273+0000 7f65a9a7c640 10 monclient(hunting): picked mon.noname-a con 0x7f65a40a8e40 addr v2:172.21.15.69:3300/0 2023-10-30T20:45:57.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.273+0000 7f65a9a7c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f65a40b13c0 0x7f65a4141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.273+0000 7f65a9a7c640 10 monclient(hunting): picked mon.noname-c con 0x7f65a40b13c0 addr v2:172.21.15.183:3300/0 2023-10-30T20:45:57.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.273+0000 7f65a9a7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65a413f500 addr v1:172.21.15.69:6789/0 2023-10-30T20:45:57.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.273+0000 7f65a9a7c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65a40b1b00 con 0x7f65a413f500 2023-10-30T20:45:57.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.273+0000 7f65a9a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:57.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.273+0000 7f65a9a7c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65a40679a0 con 0x7f65a40a8e40 2023-10-30T20:45:57.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.273+0000 7f65a9a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:57.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.273+0000 7f65a9a7c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65a40b1060 con 0x7f65a40b13c0 2023-10-30T20:45:57.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.273+0000 7f65a9a7c640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:57.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.273+0000 7f65a9a7c640 10 monclient(hunting): authenticate will time out at 2880.562346s 2023-10-30T20:45:57.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.274+0000 7f65a2ffd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f65a40b13c0 0x7f65a4141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.274+0000 7f65a37fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f65a40a8e40 0x7f65a413efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.274+0000 7f65a37fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f65a40a8e40 0x7f65a413efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50714/0 (socket says 172.21.15.69:50714) 2023-10-30T20:45:57.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.274+0000 7f65a37fe640 1 -- 172.21.15.69:0/1911508121 learned_addr learned my addr 172.21.15.69:0/1911508121 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:57.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.274+0000 7f65a37fe640 10 monclient(hunting): get_auth_request con 0x7f65a40a8e40 auth_method 0 2023-10-30T20:45:57.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.274+0000 7f65a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:57.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.274+0000 7f65a37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.274+0000 7f65a37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.274+0000 7f65a2ffd640 10 monclient(hunting): get_auth_request con 0x7f65a40b13c0 auth_method 0 2023-10-30T20:45:57.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.274+0000 7f65a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:57.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.274+0000 7f65a2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.274+0000 7f65a2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.275+0000 7f65a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:57.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.275+0000 7f65a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:57.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.275+0000 7f65a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:57.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.275+0000 7f65a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:57.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.275+0000 7f65a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:57.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.275+0000 7f65a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:57.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.276+0000 7f65a2ffd640 10 monclient(hunting): handle_auth_done global_id 21259 payload 274 2023-10-30T20:45:57.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.276+0000 7f65a2ffd640 1 -- 172.21.15.69:0/1911508121 >> v2:172.21.15.69:3300/0 conn(0x7f65a40a8e40 msgr2=0x7f65a413efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:57.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.276+0000 7f65a2ffd640 1 --2- 172.21.15.69:0/1911508121 >> v2:172.21.15.69:3300/0 conn(0x7f65a40a8e40 0x7f65a413efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.276+0000 7f65a2ffd640 1 -- 172.21.15.69:0/1911508121 >> v1:172.21.15.69:6789/0 conn(0x7f65a413f500 legacy=0x7f65a41423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.276+0000 7f65a2ffd640 10 monclient: _finish_hunting 0 2023-10-30T20:45:57.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.276+0000 7f65a2ffd640 1 monclient: found mon.noname-c 2023-10-30T20:45:57.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.276+0000 7f65a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:57.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.276+0000 7f65a2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:45:57.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.276+0000 7f65a2ffd640 1 -- 172.21.15.69:0/1911508121 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a40a8920 con 0x7f65a40b13c0 2023-10-30T20:45:57.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.279+0000 7f65a2ffd640 1 --2- 172.21.15.69:0/1911508121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a40b13c0 0x7f65a4141eb0 secure :-1 s=READY pgs=5053 cs=0 l=1 rev1=1 crypto rx=0x7f65a4067880 tx=0x7f659802b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f62be3f19e5846f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:57.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.282+0000 7f65a27fc640 1 -- 172.21.15.69:0/1911508121 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659802f050 con 0x7f65a40b13c0 2023-10-30T20:45:57.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.282+0000 7f65a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:57.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.282+0000 7f65a27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:45:57.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.282+0000 7f65a27fc640 10 monclient: dump: 2023-10-30T20:45:57.288 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:57.288 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.288 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.288 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.288 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:57.289 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.289 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:57.289 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:57.289 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:57.289 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a27fc640 10 monclient: _finish_auth 0 2023-10-30T20:45:57.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a27fc640 10 monclient: _check_auth_tickets 2023-10-30T20:45:57.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:57.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a27fc640 1 -- 172.21.15.69:0/1911508121 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6598031710 con 0x7f65a40b13c0 2023-10-30T20:45:57.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a9a7c640 5 monclient: authenticate success, global_id 21259 2023-10-30T20:45:57.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a9a7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:45:57.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:57.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a9a7c640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:57.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a27fc640 1 -- 172.21.15.69:0/1911508121 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6598031ab0 con 0x7f65a40b13c0 2023-10-30T20:45:57.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:57.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a9a7c640 4 set_mon_vals no callback set 2023-10-30T20:45:57.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:57.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a27fc640 10 monclient: dump: 2023-10-30T20:45:57.292 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:57.292 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.292 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.292 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.292 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:57.293 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.293 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:57.293 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:57.293 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:57.293 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a9a7c640 10 monclient: shutdown 2023-10-30T20:45:57.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a9a7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:57.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a9a7c640 1 -- 172.21.15.69:0/1911508121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a40b13c0 msgr2=0x7f65a4141eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a9a7c640 1 --2- 172.21.15.69:0/1911508121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a40b13c0 0x7f65a4141eb0 secure :-1 s=READY pgs=5053 cs=0 l=1 rev1=1 crypto rx=0x7f65a4067880 tx=0x7f659802b7b0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a9a7c640 1 -- 172.21.15.69:0/1911508121 shutdown_connections 2023-10-30T20:45:57.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a9a7c640 1 --2- 172.21.15.69:0/1911508121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a40b13c0 0x7f65a4141eb0 unknown :-1 s=CLOSED pgs=5053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a9a7c640 1 --2- 172.21.15.69:0/1911508121 >> v2:172.21.15.69:3300/0 conn(0x7f65a40a8e40 0x7f65a413efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a9a7c640 1 -- 172.21.15.69:0/1911508121 >> 172.21.15.69:0/1911508121 conn(0x7f65a40b7180 msgr2=0x7f65a40b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:57.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a9a7c640 1 -- 172.21.15.69:0/1911508121 shutdown_connections 2023-10-30T20:45:57.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.283+0000 7f65a9a7c640 1 -- 172.21.15.69:0/1911508121 wait complete. 2023-10-30T20:45:57.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.284+0000 7f65a9a7c640 10 monclient: build_initial_monmap 2023-10-30T20:45:57.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.284+0000 7f65a9a7c640 10 monclient: monmap: 2023-10-30T20:45:57.299 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:57.299 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:57.299 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:57.299 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:57.300 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:57.300 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:57.300 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:57.300 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:57.300 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:57.301 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 1 librados: starting msgr at 2023-10-30T20:45:57.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 1 librados: starting objecter 2023-10-30T20:45:57.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 1 Processor -- start 2023-10-30T20:45:57.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 1 -- start start 2023-10-30T20:45:57.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 1 librados: setting wanted keys 2023-10-30T20:45:57.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 1 librados: calling monclient init 2023-10-30T20:45:57.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 10 monclient: init 2023-10-30T20:45:57.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:57.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T20:45:57.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65a40a8e40 0x7f65a40ad3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 10 monclient(hunting): picked mon.noname-c con 0x7f65a40a8e40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:57.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65a41da6e0 0x7f65a41dab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 10 monclient(hunting): picked mon.noname-a con 0x7f65a41da6e0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:57.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a413c650 0x7f65a413cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 10 monclient(hunting): picked mon.noname-b con 0x7f65a413c650 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:57.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:57.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a40a9a70 con 0x7f65a41da6e0 2023-10-30T20:45:57.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:57.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a41db0a0 con 0x7f65a413c650 2023-10-30T20:45:57.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:57.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a413d010 con 0x7f65a40a8e40 2023-10-30T20:45:57.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:57.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.285+0000 7f65a9a7c640 10 monclient(hunting): authenticate will time out at 2880.574720s 2023-10-30T20:45:57.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65a41da6e0 0x7f65a41dab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a37fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65a40a8e40 0x7f65a40ad3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a3fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a413c650 0x7f65a413cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65a41da6e0 0x7f65a41dab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50720/0 (socket says 172.21.15.69:50720) 2023-10-30T20:45:57.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a2ffd640 1 -- 172.21.15.69:0/815639195 learned_addr learned my addr 172.21.15.69:0/815639195 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:57.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a2ffd640 10 monclient(hunting): get_auth_request con 0x7f65a41da6e0 auth_method 0 2023-10-30T20:45:57.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:57.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a37fe640 10 monclient(hunting): get_auth_request con 0x7f65a40a8e40 auth_method 0 2023-10-30T20:45:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a3fff640 10 monclient(hunting): get_auth_request con 0x7f65a413c650 auth_method 0 2023-10-30T20:45:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.286+0000 7f65a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.287+0000 7f65a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:57.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.287+0000 7f65a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:57.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.287+0000 7f65a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:57.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.287+0000 7f65a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:57.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.287+0000 7f65a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:57.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.287+0000 7f65a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:57.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.287+0000 7f65a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:57.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.287+0000 7f65a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:57.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.287+0000 7f65a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:57.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.289+0000 7f65a3fff640 10 monclient(hunting): handle_auth_done global_id 21262 payload 824 2023-10-30T20:45:57.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.289+0000 7f65a3fff640 1 -- 172.21.15.69:0/815639195 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65a40a8e40 msgr2=0x7f65a40ad3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.289+0000 7f65a3fff640 1 --2- 172.21.15.69:0/815639195 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65a40a8e40 0x7f65a40ad3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.289+0000 7f65a3fff640 1 -- 172.21.15.69:0/815639195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65a41da6e0 msgr2=0x7f65a41dab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.289+0000 7f65a3fff640 1 --2- 172.21.15.69:0/815639195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65a41da6e0 0x7f65a41dab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.289+0000 7f65a3fff640 10 monclient: _finish_hunting 0 2023-10-30T20:45:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.289+0000 7f65a3fff640 1 monclient: found mon.noname-b 2023-10-30T20:45:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.289+0000 7f65a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:57.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.289+0000 7f65a3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:45:57.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.289+0000 7f65a3fff640 1 -- 172.21.15.69:0/815639195 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a4142ce0 con 0x7f65a413c650 2023-10-30T20:45:57.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.292+0000 7f65a3fff640 1 --2- 172.21.15.69:0/815639195 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a413c650 0x7f65a413cad0 secure :-1 s=READY pgs=5054 cs=0 l=1 rev1=1 crypto rx=0x7f659c00c700 tx=0x7f659c00cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:57.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.292+0000 7f658bfff640 10 client.?.objecter ms_handle_connect 0x7f65a413c650 2023-10-30T20:45:57.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.292+0000 7f658bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:57.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.295+0000 7f658bfff640 1 -- 172.21.15.69:0/815639195 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c003b70 con 0x7f65a413c650 2023-10-30T20:45:57.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.295+0000 7f658bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:57.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.295+0000 7f658bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:57.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.295+0000 7f658bfff640 10 monclient: dump: 2023-10-30T20:45:57.312 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:57.312 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.313 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.313 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.313 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:57.313 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.313 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:57.313 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:57.314 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:57.314 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.295+0000 7f658bfff640 10 monclient: _finish_auth 0 2023-10-30T20:45:57.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.295+0000 7f658bfff640 10 monclient: _check_auth_tickets 2023-10-30T20:45:57.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.295+0000 7f658bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:57.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.295+0000 7f658bfff640 1 -- 172.21.15.69:0/815639195 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659c00e040 con 0x7f65a413c650 2023-10-30T20:45:57.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.295+0000 7f658bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:57.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.295+0000 7f658bfff640 1 -- 172.21.15.69:0/815639195 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c0050d0 con 0x7f65a413c650 2023-10-30T20:45:57.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.295+0000 7f65a9a7c640 5 monclient: authenticate success, global_id 21262 2023-10-30T20:45:57.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.295+0000 7f65a9a7c640 10 monclient: _renew_subs 2023-10-30T20:45:57.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.295+0000 7f65a9a7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:57.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.296+0000 7f65a9a7c640 1 -- 172.21.15.69:0/815639195 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a4144730 con 0x7f65a413c650 2023-10-30T20:45:57.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.296+0000 7f658bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:57.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.296+0000 7f658bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:57.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.295+0000 7f65a1ffb640 4 set_mon_vals no callback set 2023-10-30T20:45:57.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.296+0000 7f658bfff640 10 monclient: dump: 2023-10-30T20:45:57.316 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:57.318 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:57.318 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:57.318 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.296+0000 7f65a9a7c640 10 client.21262.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:57.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.296+0000 7f65a9a7c640 10 monclient: _renew_subs 2023-10-30T20:45:57.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.296+0000 7f65a9a7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:57.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.296+0000 7f65a9a7c640 1 -- 172.21.15.69:0/815639195 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a41433e0 con 0x7f65a413c650 2023-10-30T20:45:57.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.296+0000 7f65a9a7c640 1 librados: init done 2023-10-30T20:45:57.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.297+0000 7f65a9a7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:57.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.297+0000 7f65a9a7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:57.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.297+0000 7f65a9a7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.297+0000 7f65a9a7c640 1 -- 172.21.15.69:0/815639195 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a404e9b0 con 0x7f65a413c650 2023-10-30T20:45:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.316+0000 7f658bfff640 1 -- 172.21.15.69:0/815639195 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f659c005240 con 0x7f65a413c650 2023-10-30T20:45:57.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.316+0000 7f658bfff640 1 --2- 172.21.15.69:0/815639195 >> v2:172.21.15.69:6800/1876307353 conn(0x7f659006c840 0x7f659006ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.316+0000 7f65a37fe640 1 --2- 172.21.15.69:0/815639195 >> v2:172.21.15.69:6800/1876307353 conn(0x7f659006c840 0x7f659006ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.316+0000 7f65a37fe640 10 monclient: get_auth_request con 0x7f659006c840 auth_method 0 2023-10-30T20:45:57.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.316+0000 7f65a37fe640 1 --2- 172.21.15.69:0/815639195 >> v2:172.21.15.69:6800/1876307353 conn(0x7f659006c840 0x7f659006ed00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6594009bd0 tx=0x7f6594009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:57.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.316+0000 7f658bfff640 10 client.21262.objecter ms_handle_connect 0x7f659006c840 2023-10-30T20:45:57.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.540+0000 7f658bfff640 1 -- 172.21.15.69:0/815639195 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f659c058440 con 0x7f65a413c650 2023-10-30T20:45:57.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.540+0000 7f658bfff640 10 client.21262.objecter ms_dispatch 0x7f65a4000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:45:57.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.540+0000 7f658bfff640 3 client.21262.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:45:57.541 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.540+0000 7f658bfff640 3 client.21262.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:45:57.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.540+0000 7f658bfff640 20 client.21262.objecter dump_active .. 0 homeless 2023-10-30T20:45:57.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.549+0000 7f658bfff640 1 -- 172.21.15.69:0/815639195 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f659c05c4e0 con 0x7f65a413c650 2023-10-30T20:45:57.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.549+0000 7f658bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:57.551 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.549+0000 7f658bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:45:57.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.729+0000 7f65a9a7c640 1 -- 172.21.15.69:0/815639195 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f65a40a9220 con 0x7f659006c840 2023-10-30T20:45:57.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.729+0000 7f658bfff640 1 -- 172.21.15.69:0/815639195 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6594009900 con 0x7f659006c840 2023-10-30T20:45:57.731 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 3 to deep-scrub 2023-10-30T20:45:57.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 10 librados: watch_flush enter 2023-10-30T20:45:57.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 10 librados: watch_flush exit 2023-10-30T20:45:57.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 20 client.21262.objecter shutdown clearing up homeless session... 2023-10-30T20:45:57.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 10 client.21262.objecter successfully canceled tick 2023-10-30T20:45:57.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 1 -- 172.21.15.69:0/815639195 >> v2:172.21.15.69:6800/1876307353 conn(0x7f659006c840 msgr2=0x7f659006ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 1 --2- 172.21.15.69:0/815639195 >> v2:172.21.15.69:6800/1876307353 conn(0x7f659006c840 0x7f659006ed00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6594009bd0 tx=0x7f6594009240 comp rx=0 tx=0).stop 2023-10-30T20:45:57.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 10 monclient: shutdown 2023-10-30T20:45:57.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:57.735 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 1 -- 172.21.15.69:0/815639195 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a413c650 msgr2=0x7f65a413cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.735 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 1 --2- 172.21.15.69:0/815639195 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a413c650 0x7f65a413cad0 secure :-1 s=READY pgs=5054 cs=0 l=1 rev1=1 crypto rx=0x7f659c00c700 tx=0x7f659c00cbc0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.735 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 1 -- 172.21.15.69:0/815639195 shutdown_connections 2023-10-30T20:45:57.735 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 1 --2- 172.21.15.69:0/815639195 >> v2:172.21.15.69:6800/1876307353 conn(0x7f659006c840 0x7f659006ed00 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.735 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 1 --2- 172.21.15.69:0/815639195 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a413c650 0x7f65a413cad0 unknown :-1 s=CLOSED pgs=5054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.735 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 1 --2- 172.21.15.69:0/815639195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65a41da6e0 0x7f65a41dab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 1 --2- 172.21.15.69:0/815639195 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65a40a8e40 0x7f65a40ad3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 1 -- 172.21.15.69:0/815639195 >> 172.21.15.69:0/815639195 conn(0x7f65a40b7180 msgr2=0x7f65a413e710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:57.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 1 -- 172.21.15.69:0/815639195 shutdown_connections 2023-10-30T20:45:57.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 1 -- 172.21.15.69:0/815639195 wait complete. 2023-10-30T20:45:57.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.731+0000 7f65a9a7c640 1 librados: shutdown 2023-10-30T20:45:57.744 INFO:tasks.ceph:Scrubbing osd.4 2023-10-30T20:45:57.744 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-30T20:45:57.847 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.846+0000 7f4f1e183640 10 monclient: get_monmap_and_config 2023-10-30T20:45:57.847 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.846+0000 7f4f1e183640 10 monclient: build_initial_monmap 2023-10-30T20:45:57.848 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.846+0000 7f4f1e183640 10 monclient: monmap: 2023-10-30T20:45:57.848 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:57.848 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.848 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:57.847307+0000 2023-10-30T20:45:57.848 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:57.847307+0000 2023-10-30T20:45:57.848 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:57.849 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.849 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:57.849 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:57.849 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:57.849 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:57.849 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:57.850 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.851 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 1 Processor -- start 2023-10-30T20:45:57.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 1 -- start start 2023-10-30T20:45:57.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 10 monclient: init 2023-10-30T20:45:57.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:57.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-10-30T20:45:57.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4f180b2460 0x7f4f1813cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 10 monclient(hunting): picked mon.noname-a con 0x7f4f180b2460 addr v2:172.21.15.69:3300/0 2023-10-30T20:45:57.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f18071420 addr v1:172.21.15.69:6789/0 2023-10-30T20:45:57.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4f1813fe40 0x7f4f1814a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 10 monclient(hunting): picked mon.noname-c con 0x7f4f1813fe40 addr v2:172.21.15.183:3300/0 2023-10-30T20:45:57.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f180a87c0 con 0x7f4f18071420 2023-10-30T20:45:57.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:57.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f18074040 con 0x7f4f180b2460 2023-10-30T20:45:57.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:57.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f18070fd0 con 0x7f4f1813fe40 2023-10-30T20:45:57.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:57.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.847+0000 7f4f1e183640 10 monclient(hunting): authenticate will time out at 2881.136608s 2023-10-30T20:45:57.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.848+0000 7f4f177fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4f180b2460 0x7f4f1813cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.848+0000 7f4f17fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4f1813fe40 0x7f4f1814a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.848+0000 7f4f17fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4f1813fe40 0x7f4f1814a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:35894/0 (socket says 172.21.15.69:35894) 2023-10-30T20:45:57.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.848+0000 7f4f17fff640 1 -- 172.21.15.69:0/3347454955 learned_addr learned my addr 172.21.15.69:0/3347454955 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:57.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.848+0000 7f4f17fff640 10 monclient(hunting): get_auth_request con 0x7f4f1813fe40 auth_method 0 2023-10-30T20:45:57.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.848+0000 7f4f17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:57.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.848+0000 7f4f17fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.848+0000 7f4f17fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.848+0000 7f4f177fe640 10 monclient(hunting): get_auth_request con 0x7f4f180b2460 auth_method 0 2023-10-30T20:45:57.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.848+0000 7f4f177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:57.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.848+0000 7f4f177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.857 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.848+0000 7f4f177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.857 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.849+0000 7f4f17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:57.857 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.849+0000 7f4f17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:57.857 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.849+0000 7f4f17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:57.857 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.849+0000 7f4f177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:57.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.849+0000 7f4f177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:57.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.849+0000 7f4f177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:57.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.850+0000 7f4f17fff640 10 monclient(hunting): handle_auth_done global_id 21265 payload 274 2023-10-30T20:45:57.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.850+0000 7f4f17fff640 1 -- 172.21.15.69:0/3347454955 >> v2:172.21.15.69:3300/0 conn(0x7f4f180b2460 msgr2=0x7f4f1813cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:57.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.850+0000 7f4f17fff640 1 --2- 172.21.15.69:0/3347454955 >> v2:172.21.15.69:3300/0 conn(0x7f4f180b2460 0x7f4f1813cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.850+0000 7f4f17fff640 1 -- 172.21.15.69:0/3347454955 >> v1:172.21.15.69:6789/0 conn(0x7f4f18071420 legacy=0x7f4f1813f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.850+0000 7f4f17fff640 10 monclient: _finish_hunting 0 2023-10-30T20:45:57.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.850+0000 7f4f17fff640 1 monclient: found mon.noname-c 2023-10-30T20:45:57.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.850+0000 7f4f17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:57.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.850+0000 7f4f17fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:45:57.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.850+0000 7f4f17fff640 1 -- 172.21.15.69:0/3347454955 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f18070330 con 0x7f4f1813fe40 2023-10-30T20:45:57.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.852+0000 7f4f17fff640 1 --2- 172.21.15.69:0/3347454955 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f1813fe40 0x7f4f1814a040 secure :-1 s=READY pgs=5055 cs=0 l=1 rev1=1 crypto rx=0x7f4f1806d120 tx=0x7f4f1002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a222c863aa9b8ac3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:57.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f167fc640 1 -- 172.21.15.69:0/3347454955 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1002ca80 con 0x7f4f1813fe40 2023-10-30T20:45:57.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:57.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:45:57.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f167fc640 10 monclient: dump: 2023-10-30T20:45:57.864 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:57.864 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.864 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.864 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.864 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:57.865 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.865 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:57.865 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:57.865 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:57.865 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f167fc640 10 monclient: _finish_auth 0 2023-10-30T20:45:57.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f167fc640 10 monclient: _check_auth_tickets 2023-10-30T20:45:57.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:57.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f1e183640 5 monclient: authenticate success, global_id 21265 2023-10-30T20:45:57.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f1e183640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:45:57.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f167fc640 1 -- 172.21.15.69:0/3347454955 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1002cbf0 con 0x7f4f1813fe40 2023-10-30T20:45:57.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f167fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:57.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f1e183640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:57.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f1e183640 4 set_mon_vals no callback set 2023-10-30T20:45:57.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f167fc640 1 -- 172.21.15.69:0/3347454955 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f100343f0 con 0x7f4f1813fe40 2023-10-30T20:45:57.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:57.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:57.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f167fc640 10 monclient: dump: 2023-10-30T20:45:57.868 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:57.868 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.868 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.868 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.869 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:57.869 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.869 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:57.869 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:57.869 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:57.869 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f1e183640 10 monclient: shutdown 2023-10-30T20:45:57.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f1e183640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:57.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f1e183640 1 -- 172.21.15.69:0/3347454955 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f1813fe40 msgr2=0x7f4f1814a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.856+0000 7f4f1e183640 1 --2- 172.21.15.69:0/3347454955 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f1813fe40 0x7f4f1814a040 secure :-1 s=READY pgs=5055 cs=0 l=1 rev1=1 crypto rx=0x7f4f1806d120 tx=0x7f4f1002bee0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.857+0000 7f4f1e183640 1 -- 172.21.15.69:0/3347454955 shutdown_connections 2023-10-30T20:45:57.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.857+0000 7f4f1e183640 1 --2- 172.21.15.69:0/3347454955 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f1813fe40 0x7f4f1814a040 unknown :-1 s=CLOSED pgs=5055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.857+0000 7f4f1e183640 1 --2- 172.21.15.69:0/3347454955 >> v2:172.21.15.69:3300/0 conn(0x7f4f180b2460 0x7f4f1813cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.857+0000 7f4f1e183640 1 -- 172.21.15.69:0/3347454955 >> 172.21.15.69:0/3347454955 conn(0x7f4f180b5690 msgr2=0x7f4f180b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:57.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.857+0000 7f4f1e183640 1 -- 172.21.15.69:0/3347454955 shutdown_connections 2023-10-30T20:45:57.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.857+0000 7f4f1e183640 1 -- 172.21.15.69:0/3347454955 wait complete. 2023-10-30T20:45:57.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.857+0000 7f4f1e183640 10 monclient: build_initial_monmap 2023-10-30T20:45:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.857+0000 7f4f1e183640 10 monclient: monmap: 2023-10-30T20:45:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:57.873 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.857+0000 7f4f1e183640 1 librados: starting msgr at 2023-10-30T20:45:57.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.857+0000 7f4f1e183640 1 librados: starting objecter 2023-10-30T20:45:57.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.857+0000 7f4f1e183640 1 Processor -- start 2023-10-30T20:45:57.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 1 -- start start 2023-10-30T20:45:57.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 1 librados: setting wanted keys 2023-10-30T20:45:57.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 1 librados: calling monclient init 2023-10-30T20:45:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 10 monclient: init 2023-10-30T20:45:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T20:45:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f18071420 0x7f4f1806ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 10 monclient(hunting): picked mon.noname-c con 0x7f4f18071420 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f180b2460 0x7f4f1806c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 10 monclient(hunting): picked mon.noname-a con 0x7f4f180b2460 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:57.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f1813fe40 0x7f4f180adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 10 monclient(hunting): picked mon.noname-b con 0x7f4f1813fe40 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:57.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:57.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f180af2a0 con 0x7f4f180b2460 2023-10-30T20:45:57.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:57.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f18143a90 con 0x7f4f1813fe40 2023-10-30T20:45:57.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:57.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f180aecd0 con 0x7f4f18071420 2023-10-30T20:45:57.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:57.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.858+0000 7f4f1e183640 10 monclient(hunting): authenticate will time out at 2881.147530s 2023-10-30T20:45:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f16ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f180b2460 0x7f4f1806c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f17fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f1813fe40 0x7f4f180adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f177fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f18071420 0x7f4f1806ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f16ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f180b2460 0x7f4f1806c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50734/0 (socket says 172.21.15.69:50734) 2023-10-30T20:45:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f16ffd640 1 -- 172.21.15.69:0/1110030436 learned_addr learned my addr 172.21.15.69:0/1110030436 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f16ffd640 10 monclient(hunting): get_auth_request con 0x7f4f180b2460 auth_method 0 2023-10-30T20:45:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f16ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f16ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f17fff640 10 monclient(hunting): get_auth_request con 0x7f4f1813fe40 auth_method 0 2023-10-30T20:45:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f17fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f17fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f177fe640 10 monclient(hunting): get_auth_request con 0x7f4f18071420 auth_method 0 2023-10-30T20:45:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.859+0000 7f4f177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.860+0000 7f4f16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:57.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.860+0000 7f4f16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:57.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.860+0000 7f4f16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:57.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.860+0000 7f4f17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:57.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.860+0000 7f4f17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:57.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.860+0000 7f4f17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:57.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.860+0000 7f4f177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:57.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.860+0000 7f4f177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:57.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.860+0000 7f4f177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:57.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.862+0000 7f4f17fff640 10 monclient(hunting): handle_auth_done global_id 21268 payload 824 2023-10-30T20:45:57.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.862+0000 7f4f17fff640 1 -- 172.21.15.69:0/1110030436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f18071420 msgr2=0x7f4f1806ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.862+0000 7f4f17fff640 1 --2- 172.21.15.69:0/1110030436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f18071420 0x7f4f1806ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.862+0000 7f4f17fff640 1 -- 172.21.15.69:0/1110030436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f180b2460 msgr2=0x7f4f1806c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.862+0000 7f4f17fff640 1 --2- 172.21.15.69:0/1110030436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f180b2460 0x7f4f1806c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.862+0000 7f4f17fff640 10 monclient: _finish_hunting 0 2023-10-30T20:45:57.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.862+0000 7f4f17fff640 1 monclient: found mon.noname-b 2023-10-30T20:45:57.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.862+0000 7f4f17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:57.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.862+0000 7f4f17fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:45:57.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.862+0000 7f4f17fff640 1 -- 172.21.15.69:0/1110030436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f180ad2e0 con 0x7f4f1813fe40 2023-10-30T20:45:57.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.864+0000 7f4f17fff640 1 --2- 172.21.15.69:0/1110030436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f1813fe40 0x7f4f180adf90 secure :-1 s=READY pgs=5056 cs=0 l=1 rev1=1 crypto rx=0x7f4f10032000 tx=0x7f4f10004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:57.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.864+0000 7f4efffff640 10 client.?.objecter ms_handle_connect 0x7f4f1813fe40 2023-10-30T20:45:57.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.864+0000 7f4efffff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:57.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4efffff640 1 -- 172.21.15.69:0/1110030436 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f10009510 con 0x7f4f1813fe40 2023-10-30T20:45:57.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4efffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:57.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4efffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:57.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4efffff640 10 monclient: dump: 2023-10-30T20:45:57.886 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:57.887 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.887 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.887 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.887 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:57.887 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.887 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:57.888 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:57.888 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:57.888 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4efffff640 10 monclient: _finish_auth 0 2023-10-30T20:45:57.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4efffff640 10 monclient: _check_auth_tickets 2023-10-30T20:45:57.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4efffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:57.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4efffff640 1 -- 172.21.15.69:0/1110030436 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1003e450 con 0x7f4f1813fe40 2023-10-30T20:45:57.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4f1e183640 5 monclient: authenticate success, global_id 21268 2023-10-30T20:45:57.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4efffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:57.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4f1e183640 10 monclient: _renew_subs 2023-10-30T20:45:57.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4efffff640 1 -- 172.21.15.69:0/1110030436 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1003e850 con 0x7f4f1813fe40 2023-10-30T20:45:57.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4f1e183640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:57.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4f1e183640 1 -- 172.21.15.69:0/1110030436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1806cbb0 con 0x7f4f1813fe40 2023-10-30T20:45:57.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4efffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:57.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4efffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:57.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4f15ffb640 4 set_mon_vals no callback set 2023-10-30T20:45:57.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4efffff640 10 monclient: dump: 2023-10-30T20:45:57.891 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:57.891 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:57.891 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.891 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:57.891 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:57.892 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:57.892 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:57.892 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:57.892 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:57.892 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:57.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4f1e183640 10 client.21268.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:57.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4f1e183640 10 monclient: _renew_subs 2023-10-30T20:45:57.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4f1e183640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:57.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4f1e183640 1 -- 172.21.15.69:0/1110030436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f180ae6b0 con 0x7f4f1813fe40 2023-10-30T20:45:57.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.868+0000 7f4f1e183640 1 librados: init done 2023-10-30T20:45:57.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.869+0000 7f4f1e183640 10 client.21268.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:57.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.869+0000 7f4f1e183640 15 client.21268.objecter _session_command_op_assign -1 1 2023-10-30T20:45:57.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.869+0000 7f4f1e183640 20 client.21268.objecter _get_session osd=-1 returning homeless 2023-10-30T20:45:57.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.869+0000 7f4f1e183640 10 client.21268.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:57.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.869+0000 7f4f1e183640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:57.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.869+0000 7f4f1e183640 1 -- 172.21.15.69:0/1110030436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4f1806ff50 con 0x7f4f1813fe40 2023-10-30T20:45:57.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.887+0000 7f4efffff640 1 -- 172.21.15.69:0/1110030436 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f4f1002ca90 con 0x7f4f1813fe40 2023-10-30T20:45:57.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.887+0000 7f4efffff640 1 --2- 172.21.15.69:0/1110030436 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4f0406c810 0x7f4f0406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.887+0000 7f4f177fe640 1 --2- 172.21.15.69:0/1110030436 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4f0406c810 0x7f4f0406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.887+0000 7f4f177fe640 10 monclient: get_auth_request con 0x7f4f0406c810 auth_method 0 2023-10-30T20:45:57.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.888+0000 7f4efffff640 10 client.21268.objecter ms_handle_connect 0x7f4f0406c810 2023-10-30T20:45:57.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.888+0000 7f4f177fe640 1 --2- 172.21.15.69:0/1110030436 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4f0406c810 0x7f4f0406ecd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4f08009b40 tx=0x7f4f08009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:57.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 1 -- 172.21.15.69:0/1110030436 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f4f1007aed0 con 0x7f4f1813fe40 2023-10-30T20:45:57.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 10 client.21268.objecter ms_dispatch 0x7f4f18000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:45:57.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 3 client.21268.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:45:57.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 3 client.21268.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:45:57.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 10 client.21268.objecter checking command 1 2023-10-30T20:45:57.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 1 --2- 172.21.15.69:0/1110030436 >> v2:172.21.15.183:6800/4151811482 conn(0x7f4f04073c50 0x7f4f040760f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:57.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 20 client.21268.objecter _get_session s=0x7f4f1003e850 osd=4 3 2023-10-30T20:45:57.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 20 client.21268.objecter put_session s=0x7f4f1003e850 osd=4 3 2023-10-30T20:45:57.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 15 client.21268.objecter _session_command_op_remove -1 1 2023-10-30T20:45:57.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 20 client.21268.objecter _get_session s=0x7f4f1003e850 osd=4 3 2023-10-30T20:45:57.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 20 client.21268.objecter get_session s=0x7f4f1003e850 osd=4 3 2023-10-30T20:45:57.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 15 client.21268.objecter _session_command_op_assign 4 1 2023-10-30T20:45:57.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 20 client.21268.objecter put_session s=0x7f4f1003e850 osd=4 4 2023-10-30T20:45:57.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 10 client.21268.objecter _send_command 1 2023-10-30T20:45:57.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 1 -- 172.21.15.69:0/1110030436 --> v2:172.21.15.183:6800/4151811482 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f10009510 con 0x7f4f04073c50 2023-10-30T20:45:57.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.889+0000 7f4efffff640 20 client.21268.objecter dump_active .. 0 homeless 2023-10-30T20:45:57.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.890+0000 7f4f16ffd640 1 --2- 172.21.15.69:0/1110030436 >> v2:172.21.15.183:6800/4151811482 conn(0x7f4f04073c50 0x7f4f040760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:57.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.890+0000 7f4f16ffd640 10 monclient: get_auth_request con 0x7f4f04073c50 auth_method 0 2023-10-30T20:45:57.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.891+0000 7f4efffff640 1 -- 172.21.15.69:0/1110030436 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=904) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f10009b30 con 0x7f4f1813fe40 2023-10-30T20:45:57.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.891+0000 7f4efffff640 10 monclient: handle_get_version_reply finishing 1 version 904 2023-10-30T20:45:57.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.893+0000 7f4f16ffd640 1 --2- 172.21.15.69:0/1110030436 >> v2:172.21.15.183:6800/4151811482 conn(0x7f4f04073c50 0x7f4f040760f0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:57.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.893+0000 7f4efffff640 10 client.21268.objecter ms_handle_connect 0x7f4f04073c50 2023-10-30T20:45:57.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.915+0000 7f4efffff640 1 -- 172.21.15.69:0/1110030436 <== osd.4 v2:172.21.15.183:6800/4151811482 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f4f10009510 con 0x7f4f04073c50 2023-10-30T20:45:57.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.915+0000 7f4efffff640 10 client.21268.objecter ms_dispatch 0x7f4f18000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:45:57.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.915+0000 7f4efffff640 10 client.21268.objecter _finish_command 1 = system:0 2023-10-30T20:45:57.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.915+0000 7f4efffff640 20 client.21268.objecter put_session s=0x7f4f1003e850 osd=4 4 2023-10-30T20:45:57.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.915+0000 7f4efffff640 15 client.21268.objecter _session_command_op_remove 4 1 2023-10-30T20:45:57.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.927+0000 7f4f1e183640 10 client.21268.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T20:45:57.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.927+0000 7f4f1e183640 15 client.21268.objecter _session_command_op_assign -1 2 2023-10-30T20:45:57.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.927+0000 7f4f1e183640 20 client.21268.objecter _get_session s=0x7f4f1003e850 osd=4 3 2023-10-30T20:45:57.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.927+0000 7f4f1e183640 20 client.21268.objecter put_session s=0x7f4f1003e850 osd=4 3 2023-10-30T20:45:57.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.927+0000 7f4f1e183640 20 client.21268.objecter _get_session s=0x7f4f1003e850 osd=4 3 2023-10-30T20:45:57.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.927+0000 7f4f1e183640 15 client.21268.objecter _session_command_op_remove -1 2 2023-10-30T20:45:57.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.927+0000 7f4f1e183640 20 client.21268.objecter get_session s=0x7f4f1003e850 osd=4 3 2023-10-30T20:45:57.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.927+0000 7f4f1e183640 15 client.21268.objecter _session_command_op_assign 4 2 2023-10-30T20:45:57.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.927+0000 7f4f1e183640 20 client.21268.objecter put_session s=0x7f4f1003e850 osd=4 4 2023-10-30T20:45:57.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.927+0000 7f4f1e183640 10 client.21268.objecter _send_command 2 2023-10-30T20:45:57.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.927+0000 7f4f1e183640 1 -- 172.21.15.69:0/1110030436 --> v2:172.21.15.183:6800/4151811482 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4f180aeea0 con 0x7f4f04073c50 2023-10-30T20:45:57.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4efffff640 1 -- 172.21.15.69:0/1110030436 <== osd.4 v2:172.21.15.183:6800/4151811482 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4f0c0014c0 con 0x7f4f04073c50 2023-10-30T20:45:57.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4efffff640 10 client.21268.objecter ms_dispatch 0x7f4f18000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:45:57.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4efffff640 10 client.21268.objecter _finish_command 2 = system:0 2023-10-30T20:45:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4efffff640 20 client.21268.objecter put_session s=0x7f4f1003e850 osd=4 4 2023-10-30T20:45:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4efffff640 15 client.21268.objecter _session_command_op_remove 4 2 2023-10-30T20:45:57.943 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-10-30T20:45:57.943 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T20:45:57.943 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-10-30T20:45:57.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 10 librados: watch_flush enter 2023-10-30T20:45:57.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 10 librados: watch_flush exit 2023-10-30T20:45:57.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 10 client.21268.objecter close_session for osd.4 2023-10-30T20:45:57.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 1 -- 172.21.15.69:0/1110030436 >> v2:172.21.15.183:6800/4151811482 conn(0x7f4f04073c50 msgr2=0x7f4f040760f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 1 --2- 172.21.15.69:0/1110030436 >> v2:172.21.15.183:6800/4151811482 conn(0x7f4f04073c50 0x7f4f040760f0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:57.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 20 client.21268.objecter put_session s=0x7f4f1003e850 osd=4 1 2023-10-30T20:45:57.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 20 client.21268.objecter shutdown clearing up homeless session... 2023-10-30T20:45:57.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 10 client.21268.objecter successfully canceled tick 2023-10-30T20:45:57.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 1 -- 172.21.15.69:0/1110030436 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4f0406c810 msgr2=0x7f4f0406ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 1 --2- 172.21.15.69:0/1110030436 >> v2:172.21.15.69:6800/1876307353 conn(0x7f4f0406c810 0x7f4f0406ecd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4f08009b40 tx=0x7f4f08009190 comp rx=0 tx=0).stop 2023-10-30T20:45:57.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 10 monclient: shutdown 2023-10-30T20:45:57.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:57.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 1 -- 172.21.15.69:0/1110030436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f1813fe40 msgr2=0x7f4f180adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:57.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 1 --2- 172.21.15.69:0/1110030436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f1813fe40 0x7f4f180adf90 secure :-1 s=READY pgs=5056 cs=0 l=1 rev1=1 crypto rx=0x7f4f10032000 tx=0x7f4f10004a40 comp rx=0 tx=0).stop 2023-10-30T20:45:57.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f17fff640 1 -- 172.21.15.69:0/1110030436 reap_dead start 2023-10-30T20:45:57.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 1 -- 172.21.15.69:0/1110030436 shutdown_connections 2023-10-30T20:45:57.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 1 -- 172.21.15.69:0/1110030436 >> 172.21.15.69:0/1110030436 conn(0x7f4f180b5690 msgr2=0x7f4f180b31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:57.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 1 -- 172.21.15.69:0/1110030436 shutdown_connections 2023-10-30T20:45:57.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 1 -- 172.21.15.69:0/1110030436 wait complete. 2023-10-30T20:45:57.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:57.941+0000 7f4f1e183640 1 librados: shutdown 2023-10-30T20:45:57.953 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-30T20:45:58.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.051+0000 7f13e6f78640 10 monclient: get_monmap_and_config 2023-10-30T20:45:58.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.051+0000 7f13e6f78640 10 monclient: build_initial_monmap 2023-10-30T20:45:58.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.051+0000 7f13e6f78640 10 monclient: monmap: 2023-10-30T20:45:58.053 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:58.053 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.054 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:58.052698+0000 2023-10-30T20:45:58.054 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:58.052698+0000 2023-10-30T20:45:58.054 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:58.054 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.054 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:58.055 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:58.055 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:58.055 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:58.055 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:58.055 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.052+0000 7f13e6f78640 1 Processor -- start 2023-10-30T20:45:58.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.052+0000 7f13e6f78640 1 -- start start 2023-10-30T20:45:58.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.052+0000 7f13e6f78640 10 monclient: init 2023-10-30T20:45:58.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13e6f78640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:58.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13e6f78640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-10-30T20:45:58.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13e6f78640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f13e0071760 addr v1:172.21.15.69:6789/0 2023-10-30T20:45:58.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13e6f78640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13e0071b40 addr v1:172.21.15.183:6789/0 2023-10-30T20:45:58.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13e6f78640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f13e013e610 0x7f13e013ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13e6f78640 10 monclient(hunting): picked mon.noname-a con 0x7f13e013e610 addr v2:172.21.15.69:3300/0 2023-10-30T20:45:58.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13e6f78640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e013c260 con 0x7f13e0071760 2023-10-30T20:45:58.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13e6f78640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e0070fd0 con 0x7f13e0071b40 2023-10-30T20:45:58.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13e6f78640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:58.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13e6f78640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e0067b90 con 0x7f13e013e610 2023-10-30T20:45:58.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13e6f78640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:58.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13e6f78640 10 monclient(hunting): authenticate will time out at 2881.342001s 2023-10-30T20:45:58.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13e5515640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f13e013e610 0x7f13e013ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:58.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13dffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f13e0071b40 0x7f13e013bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.69:55314/0 (socket says 172.21.15.69:55314) 2023-10-30T20:45:58.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.053+0000 7f13dffff640 1 -- 172.21.15.69:0/2211008151 learned_addr learned my addr 172.21.15.69:0/2211008151 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:45:58.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.054+0000 7f13e5515640 10 monclient(hunting): get_auth_request con 0x7f13e013e610 auth_method 0 2023-10-30T20:45:58.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.054+0000 7f13e5515640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:58.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.054+0000 7f13e5515640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:58.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.054+0000 7f13e5515640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:58.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.054+0000 7f13e5515640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:58.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.054+0000 7f13e5515640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:58.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.054+0000 7f13e5515640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:58.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.056+0000 7f13df7fe640 1 -- 172.21.15.69:0/2211008151 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227501888 0 0) 0x7f13cc001110 con 0x7f13e0071b40 2023-10-30T20:45:58.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.056+0000 7f13df7fe640 10 monclient(hunting): _negotiate 2023-10-30T20:45:58.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.056+0000 7f13df7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:58.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.056+0000 7f13df7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:58.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.056+0000 7f13df7fe640 10 monclient(hunting): my global_id is 21271 2023-10-30T20:45:58.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.056+0000 7f13df7fe640 1 -- 172.21.15.69:0/2211008151 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d0005730 con 0x7f13e0071b40 2023-10-30T20:45:58.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.056+0000 7f13e5515640 10 monclient(hunting): handle_auth_done global_id 16299 payload 274 2023-10-30T20:45:58.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.056+0000 7f13e5515640 1 -- 172.21.15.69:0/2211008151 >> v1:172.21.15.183:6789/0 conn(0x7f13e0071b40 legacy=0x7f13e013bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.056+0000 7f13e5515640 1 -- 172.21.15.69:0/2211008151 >> v1:172.21.15.69:6789/0 conn(0x7f13e0071760 legacy=0x7f13e00acc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.056+0000 7f13e5515640 10 monclient: _finish_hunting 0 2023-10-30T20:45:58.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.056+0000 7f13e5515640 1 monclient: found mon.noname-a 2023-10-30T20:45:58.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.056+0000 7f13e5515640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:58.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.056+0000 7f13e5515640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:45:58.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.056+0000 7f13e5515640 1 -- 172.21.15.69:0/2211008151 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e0071240 con 0x7f13e013e610 2023-10-30T20:45:58.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.060+0000 7f13e5515640 1 --2- 172.21.15.69:0/2211008151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13e013e610 0x7f13e013ea70 secure :-1 s=READY pgs=3182 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009d90 tx=0x7f13d802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952e6871dedeeb85 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:58.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13df7fe640 1 -- 172.21.15.69:0/2211008151 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d802cc60 con 0x7f13e013e610 2023-10-30T20:45:58.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:58.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:45:58.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13df7fe640 10 monclient: dump: 2023-10-30T20:45:58.066 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:58.066 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.067 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.067 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.067 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:58.067 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.067 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:58.067 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:58.068 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:58.068 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13df7fe640 10 monclient: _finish_auth 0 2023-10-30T20:45:58.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13df7fe640 10 monclient: _check_auth_tickets 2023-10-30T20:45:58.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:58.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13df7fe640 1 -- 172.21.15.69:0/2211008151 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d802cdd0 con 0x7f13e013e610 2023-10-30T20:45:58.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13df7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:58.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13df7fe640 1 -- 172.21.15.69:0/2211008151 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d8034660 con 0x7f13e013e610 2023-10-30T20:45:58.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:58.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:45:58.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13df7fe640 10 monclient: dump: 2023-10-30T20:45:58.070 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:58.070 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.070 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.070 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.070 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:58.071 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.071 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:58.071 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:58.071 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:58.071 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13e6f78640 5 monclient: authenticate success, global_id 16299 2023-10-30T20:45:58.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13e6f78640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:58.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.064+0000 7f13e6f78640 4 set_mon_vals no callback set 2023-10-30T20:45:58.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.065+0000 7f13e6f78640 10 monclient: shutdown 2023-10-30T20:45:58.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.065+0000 7f13e6f78640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:58.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.065+0000 7f13e6f78640 1 -- 172.21.15.69:0/2211008151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13e013e610 msgr2=0x7f13e013ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.065+0000 7f13e6f78640 1 --2- 172.21.15.69:0/2211008151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13e013e610 0x7f13e013ea70 secure :-1 s=READY pgs=3182 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009d90 tx=0x7f13d802beb0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.065+0000 7f13e6f78640 1 -- 172.21.15.69:0/2211008151 shutdown_connections 2023-10-30T20:45:58.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.065+0000 7f13e6f78640 1 --2- 172.21.15.69:0/2211008151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13e013e610 0x7f13e013ea70 unknown :-1 s=CLOSED pgs=3182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.065+0000 7f13e6f78640 1 -- 172.21.15.69:0/2211008151 >> 172.21.15.69:0/2211008151 conn(0x7f13e0069d50 msgr2=0x7f13e006a130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:58.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.065+0000 7f13e6f78640 1 -- 172.21.15.69:0/2211008151 shutdown_connections 2023-10-30T20:45:58.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.065+0000 7f13e6f78640 1 -- 172.21.15.69:0/2211008151 wait complete. 2023-10-30T20:45:58.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.066+0000 7f13e6f78640 10 monclient: build_initial_monmap 2023-10-30T20:45:58.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.066+0000 7f13e6f78640 10 monclient: monmap: 2023-10-30T20:45:58.077 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:58.077 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:58.077 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:58.077 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:58.078 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:58.078 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:58.078 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:58.078 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:58.078 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:58.078 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.066+0000 7f13e6f78640 1 librados: starting msgr at 2023-10-30T20:45:58.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.066+0000 7f13e6f78640 1 librados: starting objecter 2023-10-30T20:45:58.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.066+0000 7f13e6f78640 1 Processor -- start 2023-10-30T20:45:58.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.066+0000 7f13e6f78640 1 -- start start 2023-10-30T20:45:58.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.066+0000 7f13e6f78640 1 librados: setting wanted keys 2023-10-30T20:45:58.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.066+0000 7f13e6f78640 1 librados: calling monclient init 2023-10-30T20:45:58.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.066+0000 7f13e6f78640 10 monclient: init 2023-10-30T20:45:58.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:58.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:45:58.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13e0071760 0x7f13e01d7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 10 monclient(hunting): picked mon.noname-b con 0x7f13e0071760 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:58.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13e013e610 0x7f13e00a8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 10 monclient(hunting): picked mon.noname-c con 0x7f13e013e610 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:58.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13e00a9980 0x7f13e00a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 10 monclient(hunting): picked mon.noname-a con 0x7f13e00a9980 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:58.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:58.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e00af190 con 0x7f13e00a9980 2023-10-30T20:45:58.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:58.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e01d7f70 con 0x7f13e0071760 2023-10-30T20:45:58.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:58.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e00ab530 con 0x7f13e013e610 2023-10-30T20:45:58.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:58.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13e6f78640 10 monclient(hunting): authenticate will time out at 2881.356247s 2023-10-30T20:45:58.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.067+0000 7f13dffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13e013e610 0x7f13e00a8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:58.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e5515640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13e00a9980 0x7f13e00a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:58.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e4d14640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13e0071760 0x7f13e01d7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:58.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e5515640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13e00a9980 0x7f13e00a9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50750/0 (socket says 172.21.15.69:50750) 2023-10-30T20:45:58.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e5515640 1 -- 172.21.15.69:0/2148645736 learned_addr learned my addr 172.21.15.69:0/2148645736 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:58.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e5515640 10 monclient(hunting): get_auth_request con 0x7f13e00a9980 auth_method 0 2023-10-30T20:45:58.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e5515640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:58.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e5515640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:58.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e5515640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:58.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13dffff640 10 monclient(hunting): get_auth_request con 0x7f13e013e610 auth_method 0 2023-10-30T20:45:58.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:58.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13dffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:58.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13dffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:58.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e4d14640 10 monclient(hunting): get_auth_request con 0x7f13e0071760 auth_method 0 2023-10-30T20:45:58.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e4d14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:58.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e4d14640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:58.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e4d14640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:58.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e5515640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:58.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e5515640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:58.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e5515640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:58.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e4d14640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:58.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e4d14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:58.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13e4d14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:58.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:58.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:58.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.068+0000 7f13dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:58.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.071+0000 7f13e4d14640 10 monclient(hunting): handle_auth_done global_id 21274 payload 824 2023-10-30T20:45:58.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.071+0000 7f13e4d14640 1 -- 172.21.15.69:0/2148645736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13e013e610 msgr2=0x7f13e00a8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.071+0000 7f13e4d14640 1 --2- 172.21.15.69:0/2148645736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13e013e610 0x7f13e00a8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.071+0000 7f13e4d14640 1 -- 172.21.15.69:0/2148645736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13e00a9980 msgr2=0x7f13e00a9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.071+0000 7f13e4d14640 1 --2- 172.21.15.69:0/2148645736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13e00a9980 0x7f13e00a9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.071+0000 7f13e4d14640 10 monclient: _finish_hunting 0 2023-10-30T20:45:58.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.071+0000 7f13e4d14640 1 monclient: found mon.noname-b 2023-10-30T20:45:58.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.071+0000 7f13e4d14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:58.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.071+0000 7f13e4d14640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:45:58.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.071+0000 7f13e4d14640 1 -- 172.21.15.69:0/2148645736 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e00a85d0 con 0x7f13e0071760 2023-10-30T20:45:58.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.071+0000 7f13e5515640 1 --2- 172.21.15.69:0/2148645736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13e00a9980 0x7f13e00a9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T20:45:58.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.073+0000 7f13e4d14640 1 --2- 172.21.15.69:0/2148645736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13e0071760 0x7f13e01d7a30 secure :-1 s=READY pgs=5058 cs=0 l=1 rev1=1 crypto rx=0x7f13d400c340 tx=0x7f13d400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:58.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.073+0000 7f13dd7fa640 10 client.?.objecter ms_handle_connect 0x7f13e0071760 2023-10-30T20:45:58.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.073+0000 7f13dd7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:58.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13dd7fa640 1 -- 172.21.15.69:0/2148645736 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d40052d0 con 0x7f13e0071760 2023-10-30T20:45:58.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:58.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13dd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:58.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13dd7fa640 10 monclient: dump: 2023-10-30T20:45:58.092 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:58.092 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.092 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.092 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.092 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:58.092 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.093 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:58.093 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:58.093 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:58.093 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13dd7fa640 10 monclient: _finish_auth 0 2023-10-30T20:45:58.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13dd7fa640 10 monclient: _check_auth_tickets 2023-10-30T20:45:58.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:58.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13dd7fa640 1 -- 172.21.15.69:0/2148645736 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d400e040 con 0x7f13e0071760 2023-10-30T20:45:58.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13e6f78640 5 monclient: authenticate success, global_id 21274 2023-10-30T20:45:58.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13dd7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:58.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13dd7fa640 1 -- 172.21.15.69:0/2148645736 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d4007260 con 0x7f13e0071760 2023-10-30T20:45:58.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13e6f78640 10 monclient: _renew_subs 2023-10-30T20:45:58.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13e6f78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:58.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13e6f78640 1 -- 172.21.15.69:0/2148645736 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e01d7510 con 0x7f13e0071760 2023-10-30T20:45:58.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:58.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13dd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:58.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13deffd640 4 set_mon_vals no callback set 2023-10-30T20:45:58.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13dd7fa640 10 monclient: dump: 2023-10-30T20:45:58.096 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:58.096 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.096 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.096 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.096 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:58.097 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.097 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:58.097 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:58.097 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:58.097 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13e6f78640 10 client.21274.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:58.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13e6f78640 10 monclient: _renew_subs 2023-10-30T20:45:58.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13e6f78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:58.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13e6f78640 1 -- 172.21.15.69:0/2148645736 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e01d70c0 con 0x7f13e0071760 2023-10-30T20:45:58.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13e6f78640 1 librados: init done 2023-10-30T20:45:58.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13e6f78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:58.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13e6f78640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:58.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.078+0000 7f13e6f78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:58.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.079+0000 7f13e6f78640 1 -- 172.21.15.69:0/2148645736 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e006ff20 con 0x7f13e0071760 2023-10-30T20:45:58.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.317+0000 7f13dd7fa640 1 -- 172.21.15.69:0/2148645736 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f13d40073d0 con 0x7f13e0071760 2023-10-30T20:45:58.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.317+0000 7f13dd7fa640 1 --2- 172.21.15.69:0/2148645736 >> v2:172.21.15.69:6800/1876307353 conn(0x7f13c406c810 0x7f13c406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.318+0000 7f13dffff640 1 --2- 172.21.15.69:0/2148645736 >> v2:172.21.15.69:6800/1876307353 conn(0x7f13c406c810 0x7f13c406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:58.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.318+0000 7f13dffff640 10 monclient: get_auth_request con 0x7f13c406c810 auth_method 0 2023-10-30T20:45:58.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.318+0000 7f13dd7fa640 10 client.21274.objecter ms_handle_connect 0x7f13c406c810 2023-10-30T20:45:58.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.318+0000 7f13dffff640 1 --2- 172.21.15.69:0/2148645736 >> v2:172.21.15.69:6800/1876307353 conn(0x7f13c406c810 0x7f13c406ecd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f13e0142be0 tx=0x7f13cc007b90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:58.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.320+0000 7f13dd7fa640 1 -- 172.21.15.69:0/2148645736 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f13d40575d0 con 0x7f13e0071760 2023-10-30T20:45:58.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.320+0000 7f13dd7fa640 10 client.21274.objecter ms_dispatch 0x7f13e0000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:45:58.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.320+0000 7f13dd7fa640 3 client.21274.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:45:58.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.320+0000 7f13dd7fa640 3 client.21274.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:45:58.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.320+0000 7f13dd7fa640 20 client.21274.objecter dump_active .. 0 homeless 2023-10-30T20:45:58.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.331+0000 7f13dd7fa640 1 -- 172.21.15.69:0/2148645736 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f13d405b670 con 0x7f13e0071760 2023-10-30T20:45:58.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.331+0000 7f13dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:58.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.331+0000 7f13dd7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:45:58.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.511+0000 7f13e6f78640 1 -- 172.21.15.69:0/2148645736 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f13e006d0b0 con 0x7f13c406c810 2023-10-30T20:45:58.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.511+0000 7f13dd7fa640 1 -- 172.21.15.69:0/2148645736 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f13cc005850 con 0x7f13c406c810 2023-10-30T20:45:58.513 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 4 to deep-scrub 2023-10-30T20:45:58.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 10 librados: watch_flush enter 2023-10-30T20:45:58.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 10 librados: watch_flush exit 2023-10-30T20:45:58.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 20 client.21274.objecter shutdown clearing up homeless session... 2023-10-30T20:45:58.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 10 client.21274.objecter successfully canceled tick 2023-10-30T20:45:58.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 1 -- 172.21.15.69:0/2148645736 >> v2:172.21.15.69:6800/1876307353 conn(0x7f13c406c810 msgr2=0x7f13c406ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 1 --2- 172.21.15.69:0/2148645736 >> v2:172.21.15.69:6800/1876307353 conn(0x7f13c406c810 0x7f13c406ecd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f13e0142be0 tx=0x7f13cc007b90 comp rx=0 tx=0).stop 2023-10-30T20:45:58.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 10 monclient: shutdown 2023-10-30T20:45:58.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:58.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 1 -- 172.21.15.69:0/2148645736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13e0071760 msgr2=0x7f13e01d7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 1 --2- 172.21.15.69:0/2148645736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13e0071760 0x7f13e01d7a30 secure :-1 s=READY pgs=5058 cs=0 l=1 rev1=1 crypto rx=0x7f13d400c340 tx=0x7f13d400c800 comp rx=0 tx=0).stop 2023-10-30T20:45:58.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 1 -- 172.21.15.69:0/2148645736 shutdown_connections 2023-10-30T20:45:58.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 1 --2- 172.21.15.69:0/2148645736 >> v2:172.21.15.69:6800/1876307353 conn(0x7f13c406c810 0x7f13c406ecd0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 1 --2- 172.21.15.69:0/2148645736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13e00a9980 0x7f13e00a9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 1 --2- 172.21.15.69:0/2148645736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13e013e610 0x7f13e00a8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 1 --2- 172.21.15.69:0/2148645736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13e0071760 0x7f13e01d7a30 unknown :-1 s=CLOSED pgs=5058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 1 -- 172.21.15.69:0/2148645736 >> 172.21.15.69:0/2148645736 conn(0x7f13e0069d50 msgr2=0x7f13e00b0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:58.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 1 -- 172.21.15.69:0/2148645736 shutdown_connections 2023-10-30T20:45:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 1 -- 172.21.15.69:0/2148645736 wait complete. 2023-10-30T20:45:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.514+0000 7f13e6f78640 1 librados: shutdown 2023-10-30T20:45:58.530 INFO:tasks.ceph:Scrubbing osd.5 2023-10-30T20:45:58.530 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-30T20:45:58.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.658+0000 7f9a8b221640 10 monclient: get_monmap_and_config 2023-10-30T20:45:58.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.658+0000 7f9a8b221640 10 monclient: build_initial_monmap 2023-10-30T20:45:58.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.658+0000 7f9a8b221640 10 monclient: monmap: 2023-10-30T20:45:58.660 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:58.661 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.661 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:58.659932+0000 2023-10-30T20:45:58.661 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:58.659932+0000 2023-10-30T20:45:58.661 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:58.661 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.662 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:58.662 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:58.662 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:58.662 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:58.662 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:58.663 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.659+0000 7f9a8b221640 1 Processor -- start 2023-10-30T20:45:58.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.659+0000 7f9a8b221640 1 -- start start 2023-10-30T20:45:58.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.659+0000 7f9a8b221640 10 monclient: init 2023-10-30T20:45:58.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.660+0000 7f9a8b221640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:58.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.660+0000 7f9a8b221640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-10-30T20:45:58.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.660+0000 7f9a8b221640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9a840b4420 0x7f9a8413ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.660+0000 7f9a8b221640 10 monclient(hunting): picked mon.noname-c con 0x7f9a840b4420 addr v2:172.21.15.183:3300/0 2023-10-30T20:45:58.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.660+0000 7f9a8b221640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9a840b12d0 0x7f9a8413ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.660+0000 7f9a8b221640 10 monclient(hunting): picked mon.noname-a con 0x7f9a840b12d0 addr v2:172.21.15.69:3300/0 2023-10-30T20:45:58.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.660+0000 7f9a8b221640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a84141910 addr v1:172.21.15.183:6789/0 2023-10-30T20:45:58.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.660+0000 7f9a8b221640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a840a9120 con 0x7f9a84141910 2023-10-30T20:45:58.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.660+0000 7f9a8b221640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:58.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.660+0000 7f9a8b221640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a84067a10 con 0x7f9a840b12d0 2023-10-30T20:45:58.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.660+0000 7f9a8b221640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:58.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.660+0000 7f9a8b221640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a840b4800 con 0x7f9a840b4420 2023-10-30T20:45:58.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.660+0000 7f9a8b221640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:58.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.660+0000 7f9a8b221640 10 monclient(hunting): authenticate will time out at 2881.949161s 2023-10-30T20:45:58.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.661+0000 7f9a88fbd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9a840b4420 0x7f9a8413ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:58.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.661+0000 7f9a88fbd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9a840b4420 0x7f9a8413ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:35918/0 (socket says 172.21.15.69:35918) 2023-10-30T20:45:58.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.661+0000 7f9a88fbd640 1 -- 172.21.15.69:0/3164178677 learned_addr learned my addr 172.21.15.69:0/3164178677 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:58.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.661+0000 7f9a88fbd640 10 monclient(hunting): get_auth_request con 0x7f9a840b4420 auth_method 0 2023-10-30T20:45:58.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.661+0000 7f9a88fbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:58.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.661+0000 7f9a88fbd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:58.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.661+0000 7f9a88fbd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:58.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.662+0000 7f9a88fbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:58.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.662+0000 7f9a88fbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:58.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.662+0000 7f9a88fbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:58.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.664+0000 7f9a88fbd640 10 monclient(hunting): handle_auth_done global_id 21277 payload 274 2023-10-30T20:45:58.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.664+0000 7f9a88fbd640 1 -- 172.21.15.69:0/3164178677 >> v2:172.21.15.69:3300/0 conn(0x7f9a840b12d0 msgr2=0x7f9a8413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:58.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.664+0000 7f9a88fbd640 1 --2- 172.21.15.69:0/3164178677 >> v2:172.21.15.69:3300/0 conn(0x7f9a840b12d0 0x7f9a8413ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.664+0000 7f9a88fbd640 1 -- 172.21.15.69:0/3164178677 >> v1:172.21.15.183:6789/0 conn(0x7f9a84141910 legacy=0x7f9a84141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.664+0000 7f9a88fbd640 10 monclient: _finish_hunting 0 2023-10-30T20:45:58.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.664+0000 7f9a88fbd640 1 monclient: found mon.noname-c 2023-10-30T20:45:58.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.664+0000 7f9a88fbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:58.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.664+0000 7f9a88fbd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:45:58.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.664+0000 7f9a88fbd640 1 -- 172.21.15.69:0/3164178677 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a840aa460 con 0x7f9a840b4420 2023-10-30T20:45:58.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.666+0000 7f9a88fbd640 1 --2- 172.21.15.69:0/3164178677 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9a840b4420 0x7f9a8413ea50 secure :-1 s=READY pgs=5060 cs=0 l=1 rev1=1 crypto rx=0x7f9a840b4180 tx=0x7f9a7802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=246f0698d8789a40 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:58.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a837fe640 1 -- 172.21.15.69:0/3164178677 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78031870 con 0x7f9a840b4420 2023-10-30T20:45:58.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:58.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:45:58.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a837fe640 10 monclient: dump: 2023-10-30T20:45:58.674 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:58.674 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.674 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.674 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.675 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:58.675 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.675 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:58.675 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:58.675 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:58.676 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a837fe640 10 monclient: _finish_auth 0 2023-10-30T20:45:58.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a837fe640 10 monclient: _check_auth_tickets 2023-10-30T20:45:58.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:58.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a837fe640 1 -- 172.21.15.69:0/3164178677 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a780319e0 con 0x7f9a840b4420 2023-10-30T20:45:58.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a837fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:58.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a837fe640 1 -- 172.21.15.69:0/3164178677 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78031d80 con 0x7f9a840b4420 2023-10-30T20:45:58.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:58.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:58.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a837fe640 10 monclient: dump: 2023-10-30T20:45:58.677 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:58.677 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.678 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.678 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.678 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:58.678 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.678 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:58.679 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:58.679 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:58.679 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a8b221640 5 monclient: authenticate success, global_id 21277 2023-10-30T20:45:58.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a8b221640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:58.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a8b221640 4 set_mon_vals no callback set 2023-10-30T20:45:58.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a8b221640 10 monclient: shutdown 2023-10-30T20:45:58.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a8b221640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:58.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a8b221640 1 -- 172.21.15.69:0/3164178677 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9a840b4420 msgr2=0x7f9a8413ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a8b221640 1 --2- 172.21.15.69:0/3164178677 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9a840b4420 0x7f9a8413ea50 secure :-1 s=READY pgs=5060 cs=0 l=1 rev1=1 crypto rx=0x7f9a840b4180 tx=0x7f9a7802b910 comp rx=0 tx=0).stop 2023-10-30T20:45:58.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a8b221640 1 -- 172.21.15.69:0/3164178677 shutdown_connections 2023-10-30T20:45:58.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a8b221640 1 --2- 172.21.15.69:0/3164178677 >> v2:172.21.15.69:3300/0 conn(0x7f9a840b12d0 0x7f9a8413ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a8b221640 1 --2- 172.21.15.69:0/3164178677 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9a840b4420 0x7f9a8413ea50 unknown :-1 s=CLOSED pgs=5060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a8b221640 1 -- 172.21.15.69:0/3164178677 >> 172.21.15.69:0/3164178677 conn(0x7f9a84065ea0 msgr2=0x7f9a840662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:58.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a8b221640 1 -- 172.21.15.69:0/3164178677 shutdown_connections 2023-10-30T20:45:58.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.672+0000 7f9a8b221640 1 -- 172.21.15.69:0/3164178677 wait complete. 2023-10-30T20:45:58.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.673+0000 7f9a8b221640 10 monclient: build_initial_monmap 2023-10-30T20:45:58.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.673+0000 7f9a8b221640 10 monclient: monmap: 2023-10-30T20:45:58.686 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:58.686 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:58.687 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:58.687 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:58.687 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:58.687 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:58.687 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:58.687 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:58.688 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:58.688 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.673+0000 7f9a8b221640 1 librados: starting msgr at 2023-10-30T20:45:58.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.673+0000 7f9a8b221640 1 librados: starting objecter 2023-10-30T20:45:58.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.673+0000 7f9a8b221640 1 Processor -- start 2023-10-30T20:45:58.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.673+0000 7f9a8b221640 1 -- start start 2023-10-30T20:45:58.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.673+0000 7f9a8b221640 1 librados: setting wanted keys 2023-10-30T20:45:58.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.673+0000 7f9a8b221640 1 librados: calling monclient init 2023-10-30T20:45:58.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.673+0000 7f9a8b221640 10 monclient: init 2023-10-30T20:45:58.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:58.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T20:45:58.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a840b12d0 0x7f9a8413e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 10 monclient(hunting): picked mon.noname-a con 0x7f9a840b12d0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:58.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9a840b4420 0x7f9a841363f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 10 monclient(hunting): picked mon.noname-b con 0x7f9a840b4420 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:58.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a84141910 0x7f9a84136930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 10 monclient(hunting): picked mon.noname-c con 0x7f9a84141910 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:58.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:58.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a840b3120 con 0x7f9a840b12d0 2023-10-30T20:45:58.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:58.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a841396a0 con 0x7f9a840b4420 2023-10-30T20:45:58.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:58.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a84140ba0 con 0x7f9a84141910 2023-10-30T20:45:58.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:58.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.674+0000 7f9a8b221640 10 monclient(hunting): authenticate will time out at 2881.963423s 2023-10-30T20:45:58.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a83fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9a840b4420 0x7f9a841363f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:58.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a897be640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a84141910 0x7f9a84136930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:58.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a83fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9a840b4420 0x7f9a841363f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:35932/0 (socket says 172.21.15.69:35932) 2023-10-30T20:45:58.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a83fff640 1 -- 172.21.15.69:0/3543294571 learned_addr learned my addr 172.21.15.69:0/3543294571 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:58.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a83fff640 10 monclient(hunting): get_auth_request con 0x7f9a840b4420 auth_method 0 2023-10-30T20:45:58.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:58.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a83fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:58.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a83fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:58.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a897be640 10 monclient(hunting): get_auth_request con 0x7f9a84141910 auth_method 0 2023-10-30T20:45:58.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a897be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:58.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a897be640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:58.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a897be640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:58.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:58.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:58.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.675+0000 7f9a83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:58.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.676+0000 7f9a897be640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:58.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.676+0000 7f9a897be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:58.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.676+0000 7f9a897be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:58.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.678+0000 7f9a83fff640 10 monclient(hunting): handle_auth_done global_id 21280 payload 824 2023-10-30T20:45:58.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.678+0000 7f9a83fff640 1 -- 172.21.15.69:0/3543294571 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a84141910 msgr2=0x7f9a84136930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.678+0000 7f9a83fff640 1 --2- 172.21.15.69:0/3543294571 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a84141910 0x7f9a84136930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.678+0000 7f9a83fff640 1 -- 172.21.15.69:0/3543294571 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a840b12d0 msgr2=0x7f9a8413e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.678+0000 7f9a83fff640 1 --2- 172.21.15.69:0/3543294571 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a840b12d0 0x7f9a8413e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.678+0000 7f9a83fff640 10 monclient: _finish_hunting 0 2023-10-30T20:45:58.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.678+0000 7f9a83fff640 1 monclient: found mon.noname-b 2023-10-30T20:45:58.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.678+0000 7f9a83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:58.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.678+0000 7f9a83fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:45:58.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.678+0000 7f9a83fff640 1 -- 172.21.15.69:0/3543294571 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a84140d70 con 0x7f9a840b4420 2023-10-30T20:45:58.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.680+0000 7f9a83fff640 1 --2- 172.21.15.69:0/3543294571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9a840b4420 0x7f9a841363f0 secure :-1 s=READY pgs=5061 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000b450 tx=0x7f9a7000b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:58.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.680+0000 7f9a817fa640 10 client.?.objecter ms_handle_connect 0x7f9a840b4420 2023-10-30T20:45:58.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.680+0000 7f9a817fa640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:58.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a817fa640 1 -- 172.21.15.69:0/3543294571 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a70011890 con 0x7f9a840b4420 2023-10-30T20:45:58.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:58.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a817fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:58.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a817fa640 10 monclient: dump: 2023-10-30T20:45:58.700 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:58.700 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.700 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.700 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:58.702 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a817fa640 10 monclient: _finish_auth 0 2023-10-30T20:45:58.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a817fa640 10 monclient: _check_auth_tickets 2023-10-30T20:45:58.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:58.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a817fa640 1 -- 172.21.15.69:0/3543294571 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a70011a00 con 0x7f9a840b4420 2023-10-30T20:45:58.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a8b221640 5 monclient: authenticate success, global_id 21280 2023-10-30T20:45:58.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a8b221640 10 monclient: _renew_subs 2023-10-30T20:45:58.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a8b221640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:58.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a8b221640 1 -- 172.21.15.69:0/3543294571 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a84139c20 con 0x7f9a840b4420 2023-10-30T20:45:58.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a817fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:58.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a817fa640 1 -- 172.21.15.69:0/3543294571 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a70011dc0 con 0x7f9a840b4420 2023-10-30T20:45:58.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:58.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a817fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:58.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a817fa640 10 monclient: dump: 2023-10-30T20:45:58.704 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:58.704 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:58.706 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:58.706 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:58.706 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a82ffd640 4 set_mon_vals no callback set 2023-10-30T20:45:58.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a8b221640 10 client.21280.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:58.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a8b221640 10 monclient: _renew_subs 2023-10-30T20:45:58.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a8b221640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:58.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a8b221640 1 -- 172.21.15.69:0/3543294571 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a84139240 con 0x7f9a840b4420 2023-10-30T20:45:58.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.684+0000 7f9a8b221640 1 librados: init done 2023-10-30T20:45:58.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.685+0000 7f9a8b221640 10 client.21280.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:58.707 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.685+0000 7f9a8b221640 15 client.21280.objecter _session_command_op_assign -1 1 2023-10-30T20:45:58.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.685+0000 7f9a8b221640 20 client.21280.objecter _get_session osd=-1 returning homeless 2023-10-30T20:45:58.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.685+0000 7f9a8b221640 10 client.21280.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:58.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.685+0000 7f9a8b221640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:58.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.685+0000 7f9a8b221640 1 -- 172.21.15.69:0/3543294571 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9a840aa740 con 0x7f9a840b4420 2023-10-30T20:45:58.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.702+0000 7f9a817fa640 1 -- 172.21.15.69:0/3543294571 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f9a7001a400 con 0x7f9a840b4420 2023-10-30T20:45:58.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.702+0000 7f9a817fa640 1 --2- 172.21.15.69:0/3543294571 >> v2:172.21.15.69:6800/1876307353 conn(0x7f9a6806c810 0x7f9a6806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.702+0000 7f9a88fbd640 1 --2- 172.21.15.69:0/3543294571 >> v2:172.21.15.69:6800/1876307353 conn(0x7f9a6806c810 0x7f9a6806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:58.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.702+0000 7f9a88fbd640 10 monclient: get_auth_request con 0x7f9a6806c810 auth_method 0 2023-10-30T20:45:58.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.703+0000 7f9a88fbd640 1 --2- 172.21.15.69:0/3543294571 >> v2:172.21.15.69:6800/1876307353 conn(0x7f9a6806c810 0x7f9a6806ecd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9a78036000 tx=0x7f9a78030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.703+0000 7f9a817fa640 10 client.21280.objecter ms_handle_connect 0x7f9a6806c810 2023-10-30T20:45:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 1 -- 172.21.15.69:0/3543294571 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f9a70058b00 con 0x7f9a840b4420 2023-10-30T20:45:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 10 client.21280.objecter ms_dispatch 0x7f9a84000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:45:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 3 client.21280.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:45:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 3 client.21280.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:45:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 10 client.21280.objecter checking command 1 2023-10-30T20:45:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 1 --2- 172.21.15.69:0/3543294571 >> v2:172.21.15.183:6812/2276632263 conn(0x7f9a68073c50 0x7f9a680760f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 20 client.21280.objecter _get_session s=0x7f9a70011dc0 osd=5 3 2023-10-30T20:45:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 20 client.21280.objecter put_session s=0x7f9a70011dc0 osd=5 3 2023-10-30T20:45:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 15 client.21280.objecter _session_command_op_remove -1 1 2023-10-30T20:45:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 20 client.21280.objecter _get_session s=0x7f9a70011dc0 osd=5 3 2023-10-30T20:45:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 20 client.21280.objecter get_session s=0x7f9a70011dc0 osd=5 3 2023-10-30T20:45:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 15 client.21280.objecter _session_command_op_assign 5 1 2023-10-30T20:45:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 20 client.21280.objecter put_session s=0x7f9a70011dc0 osd=5 4 2023-10-30T20:45:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 10 client.21280.objecter _send_command 1 2023-10-30T20:45:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 1 -- 172.21.15.69:0/3543294571 --> v2:172.21.15.183:6812/2276632263 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a70011890 con 0x7f9a68073c50 2023-10-30T20:45:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.704+0000 7f9a817fa640 20 client.21280.objecter dump_active .. 0 homeless 2023-10-30T20:45:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.705+0000 7f9a897be640 1 --2- 172.21.15.69:0/3543294571 >> v2:172.21.15.183:6812/2276632263 conn(0x7f9a68073c50 0x7f9a680760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.705+0000 7f9a897be640 10 monclient: get_auth_request con 0x7f9a68073c50 auth_method 0 2023-10-30T20:45:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.708+0000 7f9a897be640 1 --2- 172.21.15.69:0/3543294571 >> v2:172.21.15.183:6812/2276632263 conn(0x7f9a68073c50 0x7f9a680760f0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.708+0000 7f9a817fa640 10 client.21280.objecter ms_handle_connect 0x7f9a68073c50 2023-10-30T20:45:58.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.732+0000 7f9a817fa640 1 -- 172.21.15.69:0/3543294571 <== osd.5 v2:172.21.15.183:6812/2276632263 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f9a70011890 con 0x7f9a68073c50 2023-10-30T20:45:58.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.732+0000 7f9a817fa640 10 client.21280.objecter ms_dispatch 0x7f9a84000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:45:58.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.732+0000 7f9a817fa640 10 client.21280.objecter _finish_command 1 = system:0 2023-10-30T20:45:58.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.732+0000 7f9a817fa640 20 client.21280.objecter put_session s=0x7f9a70011dc0 osd=5 4 2023-10-30T20:45:58.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.732+0000 7f9a817fa640 15 client.21280.objecter _session_command_op_remove 5 1 2023-10-30T20:45:58.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.757+0000 7f9a817fa640 1 -- 172.21.15.69:0/3543294571 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=904) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a700092a0 con 0x7f9a840b4420 2023-10-30T20:45:58.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.757+0000 7f9a817fa640 10 monclient: handle_get_version_reply finishing 1 version 904 2023-10-30T20:45:58.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.757+0000 7f9a8b221640 10 client.21280.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T20:45:58.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.757+0000 7f9a8b221640 15 client.21280.objecter _session_command_op_assign -1 2 2023-10-30T20:45:58.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.757+0000 7f9a8b221640 20 client.21280.objecter _get_session s=0x7f9a70011dc0 osd=5 3 2023-10-30T20:45:58.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.757+0000 7f9a8b221640 20 client.21280.objecter put_session s=0x7f9a70011dc0 osd=5 3 2023-10-30T20:45:58.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.757+0000 7f9a8b221640 20 client.21280.objecter _get_session s=0x7f9a70011dc0 osd=5 3 2023-10-30T20:45:58.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.757+0000 7f9a8b221640 15 client.21280.objecter _session_command_op_remove -1 2 2023-10-30T20:45:58.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.757+0000 7f9a8b221640 20 client.21280.objecter get_session s=0x7f9a70011dc0 osd=5 3 2023-10-30T20:45:58.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.757+0000 7f9a8b221640 15 client.21280.objecter _session_command_op_assign 5 2 2023-10-30T20:45:58.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.757+0000 7f9a8b221640 20 client.21280.objecter put_session s=0x7f9a70011dc0 osd=5 4 2023-10-30T20:45:58.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.757+0000 7f9a8b221640 10 client.21280.objecter _send_command 2 2023-10-30T20:45:58.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.757+0000 7f9a8b221640 1 -- 172.21.15.69:0/3543294571 --> v2:172.21.15.183:6812/2276632263 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9a840a87e0 con 0x7f9a68073c50 2023-10-30T20:45:58.772 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a817fa640 1 -- 172.21.15.69:0/3543294571 <== osd.5 v2:172.21.15.183:6812/2276632263 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f9a7c001a80 con 0x7f9a68073c50 2023-10-30T20:45:58.772 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a817fa640 10 client.21280.objecter ms_dispatch 0x7f9a84000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:45:58.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a817fa640 10 client.21280.objecter _finish_command 2 = system:0 2023-10-30T20:45:58.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a817fa640 20 client.21280.objecter put_session s=0x7f9a70011dc0 osd=5 4 2023-10-30T20:45:58.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a817fa640 15 client.21280.objecter _session_command_op_remove 5 2 2023-10-30T20:45:58.773 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-10-30T20:45:58.774 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T20:45:58.774 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-10-30T20:45:58.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a8b221640 10 librados: watch_flush enter 2023-10-30T20:45:58.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a8b221640 10 librados: watch_flush exit 2023-10-30T20:45:58.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a8b221640 10 client.21280.objecter close_session for osd.5 2023-10-30T20:45:58.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a8b221640 1 -- 172.21.15.69:0/3543294571 >> v2:172.21.15.183:6812/2276632263 conn(0x7f9a68073c50 msgr2=0x7f9a680760f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a8b221640 1 --2- 172.21.15.69:0/3543294571 >> v2:172.21.15.183:6812/2276632263 conn(0x7f9a68073c50 0x7f9a680760f0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a8b221640 20 client.21280.objecter put_session s=0x7f9a70011dc0 osd=5 1 2023-10-30T20:45:58.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a8b221640 20 client.21280.objecter shutdown clearing up homeless session... 2023-10-30T20:45:58.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a8b221640 10 client.21280.objecter successfully canceled tick 2023-10-30T20:45:58.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a8b221640 1 -- 172.21.15.69:0/3543294571 >> v2:172.21.15.69:6800/1876307353 conn(0x7f9a6806c810 msgr2=0x7f9a6806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a8b221640 1 --2- 172.21.15.69:0/3543294571 >> v2:172.21.15.69:6800/1876307353 conn(0x7f9a6806c810 0x7f9a6806ecd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9a78036000 tx=0x7f9a78030000 comp rx=0 tx=0).stop 2023-10-30T20:45:58.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a8b221640 10 monclient: shutdown 2023-10-30T20:45:58.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a8b221640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:58.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a8b221640 1 -- 172.21.15.69:0/3543294571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9a840b4420 msgr2=0x7f9a841363f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a8b221640 1 --2- 172.21.15.69:0/3543294571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9a840b4420 0x7f9a841363f0 secure :-1 s=READY pgs=5061 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000b450 tx=0x7f9a7000b910 comp rx=0 tx=0).stop 2023-10-30T20:45:58.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.771+0000 7f9a897be640 1 -- 172.21.15.69:0/3543294571 reap_dead start 2023-10-30T20:45:58.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.772+0000 7f9a8b221640 1 -- 172.21.15.69:0/3543294571 shutdown_connections 2023-10-30T20:45:58.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.772+0000 7f9a8b221640 1 -- 172.21.15.69:0/3543294571 >> 172.21.15.69:0/3543294571 conn(0x7f9a84065ea0 msgr2=0x7f9a840b7210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:58.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.772+0000 7f9a8b221640 1 -- 172.21.15.69:0/3543294571 shutdown_connections 2023-10-30T20:45:58.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.772+0000 7f9a8b221640 1 -- 172.21.15.69:0/3543294571 wait complete. 2023-10-30T20:45:58.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.772+0000 7f9a8b221640 1 librados: shutdown 2023-10-30T20:45:58.783 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-30T20:45:58.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.881+0000 7fb24fffb640 10 monclient: get_monmap_and_config 2023-10-30T20:45:58.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.881+0000 7fb24fffb640 10 monclient: build_initial_monmap 2023-10-30T20:45:58.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.881+0000 7fb24fffb640 10 monclient: monmap: 2023-10-30T20:45:58.883 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:58.883 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.883 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:58.882255+0000 2023-10-30T20:45:58.883 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:58.882255+0000 2023-10-30T20:45:58.884 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:58.884 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.884 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:58.884 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:58.884 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:58.885 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:58.885 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:58.885 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.881+0000 7fb24fffb640 1 Processor -- start 2023-10-30T20:45:58.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.882+0000 7fb24fffb640 1 -- start start 2023-10-30T20:45:58.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.882+0000 7fb24fffb640 10 monclient: init 2023-10-30T20:45:58.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.882+0000 7fb24fffb640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:58.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.882+0000 7fb24fffb640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2023-10-30T20:45:58.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.882+0000 7fb24fffb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb2480b4350 0x7fb2480a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.882+0000 7fb24fffb640 10 monclient(hunting): picked mon.noname-c con 0x7fb2480b4350 addr v2:172.21.15.183:3300/0 2023-10-30T20:45:58.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.882+0000 7fb24fffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb24813eb50 addr v1:172.21.15.183:6789/0 2023-10-30T20:45:58.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.882+0000 7fb24fffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb248144f50 addr v1:172.21.15.69:6789/0 2023-10-30T20:45:58.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.882+0000 7fb24fffb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24813f6b0 con 0x7fb248144f50 2023-10-30T20:45:58.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.882+0000 7fb24fffb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2480aa6e0 con 0x7fb24813eb50 2023-10-30T20:45:58.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.882+0000 7fb24fffb640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:58.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.882+0000 7fb24fffb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2480679f0 con 0x7fb2480b4350 2023-10-30T20:45:58.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.882+0000 7fb24fffb640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:58.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.882+0000 7fb24fffb640 10 monclient(hunting): authenticate will time out at 2882.171466s 2023-10-30T20:45:58.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.883+0000 7fb24e598640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb248144f50 0x7fb2481453a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:38970/0 (socket says 172.21.15.69:38970) 2023-10-30T20:45:58.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.883+0000 7fb24e598640 1 -- 172.21.15.69:0/4218611034 learned_addr learned my addr 172.21.15.69:0/4218611034 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:45:58.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.883+0000 7fb24dd97640 1 --2- 172.21.15.69:0/4218611034 >> v2:172.21.15.183:3300/0 conn(0x7fb2480b4350 0x7fb2480a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:58.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.884+0000 7fb24dd97640 10 monclient(hunting): get_auth_request con 0x7fb2480b4350 auth_method 0 2023-10-30T20:45:58.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.884+0000 7fb24dd97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:58.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.884+0000 7fb24dd97640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:58.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.884+0000 7fb24dd97640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:58.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.884+0000 7fb24dd97640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:58.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.884+0000 7fb24dd97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:58.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.884+0000 7fb24dd97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:58.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.885+0000 7fb24cd95640 1 -- 172.21.15.69:0/4218611034 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156320250 0 0) 0x7fb24813f6b0 con 0x7fb248144f50 2023-10-30T20:45:58.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.885+0000 7fb24cd95640 10 monclient(hunting): _negotiate 2023-10-30T20:45:58.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.885+0000 7fb24cd95640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:58.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.885+0000 7fb24cd95640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:58.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.885+0000 7fb24cd95640 10 monclient(hunting): my global_id is 16308 2023-10-30T20:45:58.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.885+0000 7fb24cd95640 1 -- 172.21.15.69:0/4218611034 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb234003520 con 0x7fb248144f50 2023-10-30T20:45:58.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.886+0000 7fb24dd97640 10 monclient(hunting): handle_auth_done global_id 21283 payload 274 2023-10-30T20:45:58.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.886+0000 7fb24dd97640 1 -- 172.21.15.69:0/4218611034 >> v1:172.21.15.183:6789/0 conn(0x7fb24813eb50 legacy=0x7fb24813efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.886+0000 7fb24dd97640 1 -- 172.21.15.69:0/4218611034 >> v1:172.21.15.69:6789/0 conn(0x7fb248144f50 legacy=0x7fb2481453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.886+0000 7fb24dd97640 10 monclient: _finish_hunting 0 2023-10-30T20:45:58.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.886+0000 7fb24dd97640 1 monclient: found mon.noname-c 2023-10-30T20:45:58.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.886+0000 7fb24dd97640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:58.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.886+0000 7fb24dd97640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:45:58.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.886+0000 7fb24dd97640 1 -- 172.21.15.69:0/4218611034 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2480a8910 con 0x7fb2480b4350 2023-10-30T20:45:58.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.890+0000 7fb24dd97640 1 --2- 172.21.15.69:0/4218611034 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2480b4350 0x7fb2480a83d0 secure :-1 s=READY pgs=5063 cs=0 l=1 rev1=1 crypto rx=0x7fb2400097a0 tx=0x7fb24002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ac7f0ff0e1f8a6d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:58.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.894+0000 7fb24cd95640 1 -- 172.21.15.69:0/4218611034 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2400315a0 con 0x7fb2480b4350 2023-10-30T20:45:58.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.894+0000 7fb24cd95640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:58.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.894+0000 7fb24cd95640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:45:58.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.894+0000 7fb24cd95640 10 monclient: dump: 2023-10-30T20:45:58.898 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:58.899 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.899 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.899 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.899 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:58.899 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.899 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:58.900 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:58.900 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:58.900 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.894+0000 7fb24cd95640 10 monclient: _finish_auth 0 2023-10-30T20:45:58.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.894+0000 7fb24cd95640 10 monclient: _check_auth_tickets 2023-10-30T20:45:58.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.894+0000 7fb24cd95640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:58.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.894+0000 7fb24cd95640 1 -- 172.21.15.69:0/4218611034 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb240031710 con 0x7fb2480b4350 2023-10-30T20:45:58.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.894+0000 7fb24cd95640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:58.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.894+0000 7fb24cd95640 1 -- 172.21.15.69:0/4218611034 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb240031ab0 con 0x7fb2480b4350 2023-10-30T20:45:58.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.894+0000 7fb24cd95640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:58.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.894+0000 7fb24cd95640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:58.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.894+0000 7fb24cd95640 10 monclient: dump: 2023-10-30T20:45:58.902 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:58.902 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.902 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.902 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.902 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:58.903 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.903 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:58.903 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:58.903 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:58.903 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.895+0000 7fb24fffb640 5 monclient: authenticate success, global_id 21283 2023-10-30T20:45:58.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.895+0000 7fb24fffb640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:58.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.895+0000 7fb24fffb640 4 set_mon_vals no callback set 2023-10-30T20:45:58.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.895+0000 7fb24fffb640 10 monclient: shutdown 2023-10-30T20:45:58.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.895+0000 7fb24fffb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:58.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.895+0000 7fb24fffb640 1 -- 172.21.15.69:0/4218611034 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2480b4350 msgr2=0x7fb2480a83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.895+0000 7fb24fffb640 1 --2- 172.21.15.69:0/4218611034 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2480b4350 0x7fb2480a83d0 secure :-1 s=READY pgs=5063 cs=0 l=1 rev1=1 crypto rx=0x7fb2400097a0 tx=0x7fb24002b840 comp rx=0 tx=0).stop 2023-10-30T20:45:58.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.895+0000 7fb24fffb640 1 -- 172.21.15.69:0/4218611034 shutdown_connections 2023-10-30T20:45:58.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.895+0000 7fb24fffb640 1 --2- 172.21.15.69:0/4218611034 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2480b4350 0x7fb2480a83d0 unknown :-1 s=CLOSED pgs=5063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.895+0000 7fb24fffb640 1 -- 172.21.15.69:0/4218611034 >> 172.21.15.69:0/4218611034 conn(0x7fb248065ea0 msgr2=0x7fb2480662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:58.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.895+0000 7fb24fffb640 1 -- 172.21.15.69:0/4218611034 shutdown_connections 2023-10-30T20:45:58.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.895+0000 7fb24fffb640 1 -- 172.21.15.69:0/4218611034 wait complete. 2023-10-30T20:45:58.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.896+0000 7fb24fffb640 10 monclient: build_initial_monmap 2023-10-30T20:45:58.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.896+0000 7fb24fffb640 10 monclient: monmap: 2023-10-30T20:45:58.906 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:58.906 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:58.906 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:58.907 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:58.907 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:58.907 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:58.907 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:58.907 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:58.907 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:58.908 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.896+0000 7fb24fffb640 1 librados: starting msgr at 2023-10-30T20:45:58.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.896+0000 7fb24fffb640 1 librados: starting objecter 2023-10-30T20:45:58.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.896+0000 7fb24fffb640 1 Processor -- start 2023-10-30T20:45:58.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.896+0000 7fb24fffb640 1 -- start start 2023-10-30T20:45:58.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.896+0000 7fb24fffb640 1 librados: setting wanted keys 2023-10-30T20:45:58.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.896+0000 7fb24fffb640 1 librados: calling monclient init 2023-10-30T20:45:58.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.896+0000 7fb24fffb640 10 monclient: init 2023-10-30T20:45:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T20:45:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2480b4350 0x7fb2480b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 10 monclient(hunting): picked mon.noname-a con 0x7fb2480b4350 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb24813eb50 0x7fb2481d6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 10 monclient(hunting): picked mon.noname-c con 0x7fb24813eb50 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:58.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb248144f50 0x7fb248143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:58.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 10 monclient(hunting): picked mon.noname-b con 0x7fb248144f50 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:58.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:58.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb248145380 con 0x7fb2480b4350 2023-10-30T20:45:58.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:58.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2481d4360 con 0x7fb248144f50 2023-10-30T20:45:58.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:58.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2481d66b0 con 0x7fb24813eb50 2023-10-30T20:45:58.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:58.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.897+0000 7fb24fffb640 10 monclient(hunting): authenticate will time out at 2882.186359s 2023-10-30T20:45:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24dd97640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2480b4350 0x7fb2480b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24d596640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb24813eb50 0x7fb2481d6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24e598640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb248144f50 0x7fb248143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24dd97640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2480b4350 0x7fb2480b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50792/0 (socket says 172.21.15.69:50792) 2023-10-30T20:45:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24dd97640 1 -- 172.21.15.69:0/3655818124 learned_addr learned my addr 172.21.15.69:0/3655818124 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24dd97640 10 monclient(hunting): get_auth_request con 0x7fb2480b4350 auth_method 0 2023-10-30T20:45:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24dd97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24dd97640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24dd97640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24d596640 10 monclient(hunting): get_auth_request con 0x7fb24813eb50 auth_method 0 2023-10-30T20:45:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24d596640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24d596640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:58.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24d596640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:58.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24e598640 10 monclient(hunting): get_auth_request con 0x7fb248144f50 auth_method 0 2023-10-30T20:45:58.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24e598640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:58.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24e598640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:58.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24e598640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:58.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24dd97640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:58.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24dd97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:58.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.898+0000 7fb24dd97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:58.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.899+0000 7fb24d596640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:58.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.899+0000 7fb24d596640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:58.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.899+0000 7fb24d596640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:58.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.899+0000 7fb24e598640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:58.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.899+0000 7fb24e598640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:58.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.899+0000 7fb24e598640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:58.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.901+0000 7fb24e598640 10 monclient(hunting): handle_auth_done global_id 21289 payload 824 2023-10-30T20:45:58.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.901+0000 7fb24e598640 1 -- 172.21.15.69:0/3655818124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb24813eb50 msgr2=0x7fb2481d6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.901+0000 7fb24e598640 1 --2- 172.21.15.69:0/3655818124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb24813eb50 0x7fb2481d6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.901+0000 7fb24e598640 1 -- 172.21.15.69:0/3655818124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2480b4350 msgr2=0x7fb2480b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:58.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.901+0000 7fb24e598640 1 --2- 172.21.15.69:0/3655818124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2480b4350 0x7fb2480b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:58.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.901+0000 7fb24e598640 10 monclient: _finish_hunting 0 2023-10-30T20:45:58.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.901+0000 7fb24e598640 1 monclient: found mon.noname-b 2023-10-30T20:45:58.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.901+0000 7fb24e598640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:58.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.901+0000 7fb24e598640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:45:58.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.901+0000 7fb24e598640 1 -- 172.21.15.69:0/3655818124 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2481d68c0 con 0x7fb248144f50 2023-10-30T20:45:58.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.903+0000 7fb24e598640 1 --2- 172.21.15.69:0/3655818124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb248144f50 0x7fb248143ff0 secure :-1 s=READY pgs=5064 cs=0 l=1 rev1=1 crypto rx=0x7fb24400cad0 tx=0x7fb24400ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:58.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.903+0000 7fb23e7fc640 10 client.?.objecter ms_handle_connect 0x7fb248144f50 2023-10-30T20:45:58.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.903+0000 7fb23e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:58.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb23e7fc640 1 -- 172.21.15.69:0/3655818124 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24400ab70 con 0x7fb248144f50 2023-10-30T20:45:58.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb23e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:58.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb23e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:58.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb23e7fc640 10 monclient: dump: 2023-10-30T20:45:58.922 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:58.922 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.922 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.923 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.923 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:58.923 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.923 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:58.923 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:58.924 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:58.924 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb23e7fc640 10 monclient: _finish_auth 0 2023-10-30T20:45:58.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb23e7fc640 10 monclient: _check_auth_tickets 2023-10-30T20:45:58.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb23e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:58.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb24fffb640 5 monclient: authenticate success, global_id 21289 2023-10-30T20:45:58.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb23e7fc640 1 -- 172.21.15.69:0/3655818124 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24400e040 con 0x7fb248144f50 2023-10-30T20:45:58.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb23e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:58.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb23e7fc640 1 -- 172.21.15.69:0/3655818124 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244012400 con 0x7fb248144f50 2023-10-30T20:45:58.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb24fffb640 10 monclient: _renew_subs 2023-10-30T20:45:58.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb24fffb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb24fffb640 1 -- 172.21.15.69:0/3655818124 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2481434b0 con 0x7fb248144f50 2023-10-30T20:45:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb23e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb23e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb23ffff640 4 set_mon_vals no callback set 2023-10-30T20:45:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb23e7fc640 10 monclient: dump: 2023-10-30T20:45:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:58.928 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb24fffb640 10 client.21289.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb24fffb640 10 monclient: _renew_subs 2023-10-30T20:45:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb24fffb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:58.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb24fffb640 1 -- 172.21.15.69:0/3655818124 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2481d7500 con 0x7fb248144f50 2023-10-30T20:45:58.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.907+0000 7fb24fffb640 1 librados: init done 2023-10-30T20:45:58.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.908+0000 7fb24fffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:58.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.908+0000 7fb24fffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:58.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.908+0000 7fb24fffb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:58.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:58.908+0000 7fb24fffb640 1 -- 172.21.15.69:0/3655818124 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2481d6ac0 con 0x7fb248144f50 2023-10-30T20:45:59.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.146+0000 7fb23e7fc640 1 -- 172.21.15.69:0/3655818124 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fb244007070 con 0x7fb248144f50 2023-10-30T20:45:59.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.147+0000 7fb23e7fc640 1 --2- 172.21.15.69:0/3655818124 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb22c06c810 0x7fb22c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.147+0000 7fb24dd97640 1 --2- 172.21.15.69:0/3655818124 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb22c06c810 0x7fb22c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.150 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.147+0000 7fb24dd97640 10 monclient: get_auth_request con 0x7fb22c06c810 auth_method 0 2023-10-30T20:45:59.150 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.147+0000 7fb23e7fc640 10 client.21289.objecter ms_handle_connect 0x7fb22c06c810 2023-10-30T20:45:59.150 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.147+0000 7fb24dd97640 1 --2- 172.21.15.69:0/3655818124 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb22c06c810 0x7fb22c06ecd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb24002b880 tx=0x7fb24003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:59.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.151+0000 7fb23e7fc640 1 -- 172.21.15.69:0/3655818124 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7fb244058110 con 0x7fb248144f50 2023-10-30T20:45:59.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.151+0000 7fb23e7fc640 10 client.21289.objecter ms_dispatch 0x7fb248000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:45:59.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.151+0000 7fb23e7fc640 3 client.21289.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:45:59.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.151+0000 7fb23e7fc640 3 client.21289.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:45:59.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.151+0000 7fb23e7fc640 20 client.21289.objecter dump_active .. 0 homeless 2023-10-30T20:45:59.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.161+0000 7fb23e7fc640 1 -- 172.21.15.69:0/3655818124 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb24405c1b0 con 0x7fb248144f50 2023-10-30T20:45:59.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.161+0000 7fb23e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:59.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.161+0000 7fb23e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:45:59.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.340+0000 7fb24fffb640 1 -- 172.21.15.69:0/3655818124 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2481434b0 con 0x7fb22c06c810 2023-10-30T20:45:59.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.341+0000 7fb23e7fc640 1 -- 172.21.15.69:0/3655818124 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb2481434b0 con 0x7fb22c06c810 2023-10-30T20:45:59.343 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 5 to deep-scrub 2023-10-30T20:45:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.342+0000 7fb24fffb640 10 librados: watch_flush enter 2023-10-30T20:45:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 10 librados: watch_flush exit 2023-10-30T20:45:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 20 client.21289.objecter shutdown clearing up homeless session... 2023-10-30T20:45:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 10 client.21289.objecter successfully canceled tick 2023-10-30T20:45:59.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 1 -- 172.21.15.69:0/3655818124 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb22c06c810 msgr2=0x7fb22c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:59.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 1 --2- 172.21.15.69:0/3655818124 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb22c06c810 0x7fb22c06ecd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb24002b880 tx=0x7fb24003d040 comp rx=0 tx=0).stop 2023-10-30T20:45:59.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 10 monclient: shutdown 2023-10-30T20:45:59.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:59.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 1 -- 172.21.15.69:0/3655818124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb248144f50 msgr2=0x7fb248143ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:59.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 1 --2- 172.21.15.69:0/3655818124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb248144f50 0x7fb248143ff0 secure :-1 s=READY pgs=5064 cs=0 l=1 rev1=1 crypto rx=0x7fb24400cad0 tx=0x7fb24400ced0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 1 -- 172.21.15.69:0/3655818124 shutdown_connections 2023-10-30T20:45:59.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 1 --2- 172.21.15.69:0/3655818124 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb22c06c810 0x7fb22c06ecd0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 1 --2- 172.21.15.69:0/3655818124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb248144f50 0x7fb248143ff0 unknown :-1 s=CLOSED pgs=5064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 1 --2- 172.21.15.69:0/3655818124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb24813eb50 0x7fb2481d6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 1 --2- 172.21.15.69:0/3655818124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2480b4350 0x7fb2480b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 1 -- 172.21.15.69:0/3655818124 >> 172.21.15.69:0/3655818124 conn(0x7fb248065ea0 msgr2=0x7fb2480737b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:59.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 1 -- 172.21.15.69:0/3655818124 shutdown_connections 2023-10-30T20:45:59.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 1 -- 172.21.15.69:0/3655818124 wait complete. 2023-10-30T20:45:59.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.343+0000 7fb24fffb640 1 librados: shutdown 2023-10-30T20:45:59.355 INFO:tasks.ceph:Scrubbing osd.6 2023-10-30T20:45:59.356 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-30T20:45:59.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.469+0000 7fb0490cf640 10 monclient: get_monmap_and_config 2023-10-30T20:45:59.471 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.469+0000 7fb0490cf640 10 monclient: build_initial_monmap 2023-10-30T20:45:59.471 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.469+0000 7fb0490cf640 10 monclient: monmap: 2023-10-30T20:45:59.471 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:59.471 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:59.472 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:59.470469+0000 2023-10-30T20:45:59.472 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:59.470469+0000 2023-10-30T20:45:59.472 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:59.472 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:59.472 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:59.473 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:59.473 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:59.473 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:59.473 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:59.474 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:59.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 1 Processor -- start 2023-10-30T20:45:59.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 1 -- start start 2023-10-30T20:45:59.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 10 monclient: init 2023-10-30T20:45:59.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:59.476 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2023-10-30T20:45:59.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0440b4420 0x7fb04413ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 10 monclient(hunting): picked mon.noname-b con 0x7fb0440b4420 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:59.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb0440b12d0 0x7fb04413ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 10 monclient(hunting): picked mon.noname-a con 0x7fb0440b12d0 addr v2:172.21.15.69:3300/0 2023-10-30T20:45:59.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb044141910 0x7fb044141da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 10 monclient(hunting): picked mon.noname-c con 0x7fb044141910 addr v2:172.21.15.183:3300/0 2023-10-30T20:45:59.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:59.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb044067a10 con 0x7fb0440b12d0 2023-10-30T20:45:59.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:59.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0440b4800 con 0x7fb044141910 2023-10-30T20:45:59.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:59.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440a9120 con 0x7fb0440b4420 2023-10-30T20:45:59.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:59.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.470+0000 7fb0490cf640 10 monclient(hunting): authenticate will time out at 2882.759794s 2023-10-30T20:45:59.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.471+0000 7fb042d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0440b4420 0x7fb04413ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.471+0000 7fb03bfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb0440b12d0 0x7fb04413ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.471+0000 7fb04359e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb044141910 0x7fb044141da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb042d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0440b4420 0x7fb04413ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41858/0 (socket says 172.21.15.69:41858) 2023-10-30T20:45:59.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb042d9d640 1 -- 172.21.15.69:0/1181360351 learned_addr learned my addr 172.21.15.69:0/1181360351 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:59.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb042d9d640 10 monclient(hunting): get_auth_request con 0x7fb0440b4420 auth_method 0 2023-10-30T20:45:59.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb042d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:59.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb042d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:59.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb042d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:59.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb03bfff640 10 monclient(hunting): get_auth_request con 0x7fb0440b12d0 auth_method 0 2023-10-30T20:45:59.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb03bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:59.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb03bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:59.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb03bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:59.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb04359e640 10 monclient(hunting): get_auth_request con 0x7fb044141910 auth_method 0 2023-10-30T20:45:59.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb04359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:59.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb04359e640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:59.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb04359e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:59.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb042d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:59.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb042d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:59.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb042d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:59.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb04359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:59.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb04359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:59.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb04359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:59.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb03bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:59.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb03bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:59.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.472+0000 7fb03bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:59.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.474+0000 7fb04359e640 10 monclient(hunting): handle_auth_done global_id 21292 payload 274 2023-10-30T20:45:59.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.474+0000 7fb04359e640 1 -- 172.21.15.69:0/1181360351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0440b4420 msgr2=0x7fb04413ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:59.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.474+0000 7fb04359e640 1 --2- 172.21.15.69:0/1181360351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0440b4420 0x7fb04413ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.474+0000 7fb04359e640 1 -- 172.21.15.69:0/1181360351 >> v2:172.21.15.69:3300/0 conn(0x7fb0440b12d0 msgr2=0x7fb04413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:59.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.474+0000 7fb04359e640 1 --2- 172.21.15.69:0/1181360351 >> v2:172.21.15.69:3300/0 conn(0x7fb0440b12d0 0x7fb04413ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.474+0000 7fb04359e640 10 monclient: _finish_hunting 0 2023-10-30T20:45:59.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.474+0000 7fb04359e640 1 monclient: found mon.noname-c 2023-10-30T20:45:59.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.474+0000 7fb04359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:59.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.474+0000 7fb04359e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:45:59.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.474+0000 7fb04359e640 1 -- 172.21.15.69:0/1181360351 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0440aa460 con 0x7fb044141910 2023-10-30T20:45:59.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.476+0000 7fb04359e640 1 --2- 172.21.15.69:0/1181360351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb044141910 0x7fb044141da0 secure :-1 s=READY pgs=5065 cs=0 l=1 rev1=1 crypto rx=0x7fb0440b3510 tx=0x7fb03c02bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2411f2179c16826 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:59.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb04259c640 1 -- 172.21.15.69:0/1181360351 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c02cb40 con 0x7fb044141910 2023-10-30T20:45:59.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb04259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:59.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb04259c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:45:59.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb04259c640 10 monclient: dump: 2023-10-30T20:45:59.497 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:59.497 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:59.498 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.498 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.498 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:59.498 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:59.498 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:59.499 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb04259c640 10 monclient: _finish_auth 0 2023-10-30T20:45:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb04259c640 10 monclient: _check_auth_tickets 2023-10-30T20:45:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb04259c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb04259c640 1 -- 172.21.15.69:0/1181360351 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c02ccb0 con 0x7fb044141910 2023-10-30T20:45:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb0490cf640 5 monclient: authenticate success, global_id 21292 2023-10-30T20:45:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb0490cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:45:59.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb04259c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:59.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb0490cf640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:59.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb0490cf640 4 set_mon_vals no callback set 2023-10-30T20:45:59.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb04259c640 1 -- 172.21.15.69:0/1181360351 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c034660 con 0x7fb044141910 2023-10-30T20:45:59.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb04259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:59.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb04259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:59.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.479+0000 7fb04259c640 10 monclient: dump: 2023-10-30T20:45:59.502 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:59.502 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:59.502 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.502 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.503 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:59.503 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:59.503 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:59.503 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:59.504 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:59.504 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:59.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.480+0000 7fb0490cf640 10 monclient: shutdown 2023-10-30T20:45:59.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.480+0000 7fb0490cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:59.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.480+0000 7fb0490cf640 1 -- 172.21.15.69:0/1181360351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb044141910 msgr2=0x7fb044141da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.480+0000 7fb0490cf640 1 --2- 172.21.15.69:0/1181360351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb044141910 0x7fb044141da0 secure :-1 s=READY pgs=5065 cs=0 l=1 rev1=1 crypto rx=0x7fb0440b3510 tx=0x7fb03c02bd90 comp rx=0 tx=0).stop 2023-10-30T20:45:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.480+0000 7fb0490cf640 1 -- 172.21.15.69:0/1181360351 shutdown_connections 2023-10-30T20:45:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.480+0000 7fb0490cf640 1 --2- 172.21.15.69:0/1181360351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb044141910 0x7fb044141da0 unknown :-1 s=CLOSED pgs=5065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.480+0000 7fb0490cf640 1 --2- 172.21.15.69:0/1181360351 >> v2:172.21.15.69:3300/0 conn(0x7fb0440b12d0 0x7fb04413ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.480+0000 7fb0490cf640 1 --2- 172.21.15.69:0/1181360351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0440b4420 0x7fb04413ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.480+0000 7fb0490cf640 1 -- 172.21.15.69:0/1181360351 >> 172.21.15.69:0/1181360351 conn(0x7fb044065ea0 msgr2=0x7fb0440662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.481+0000 7fb0490cf640 1 -- 172.21.15.69:0/1181360351 shutdown_connections 2023-10-30T20:45:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.481+0000 7fb0490cf640 1 -- 172.21.15.69:0/1181360351 wait complete. 2023-10-30T20:45:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.482+0000 7fb0490cf640 10 monclient: build_initial_monmap 2023-10-30T20:45:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.482+0000 7fb0490cf640 10 monclient: monmap: 2023-10-30T20:45:59.507 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:59.507 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:59.507 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:59.507 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:59.507 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:59.508 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:59.508 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:59.508 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:59.508 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:59.508 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:59.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.482+0000 7fb0490cf640 1 librados: starting msgr at 2023-10-30T20:45:59.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.482+0000 7fb0490cf640 1 librados: starting objecter 2023-10-30T20:45:59.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.482+0000 7fb0490cf640 1 Processor -- start 2023-10-30T20:45:59.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.482+0000 7fb0490cf640 1 -- start start 2023-10-30T20:45:59.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.482+0000 7fb0490cf640 1 librados: setting wanted keys 2023-10-30T20:45:59.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.482+0000 7fb0490cf640 1 librados: calling monclient init 2023-10-30T20:45:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.482+0000 7fb0490cf640 10 monclient: init 2023-10-30T20:45:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T20:45:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0440b12d0 0x7fb0441da1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 10 monclient(hunting): picked mon.noname-c con 0x7fb0440b12d0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:59.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0440b4420 0x7fb0441da6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 10 monclient(hunting): picked mon.noname-b con 0x7fb0440b4420 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:59.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb044141910 0x7fb0441d6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 10 monclient(hunting): picked mon.noname-a con 0x7fb044141910 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:59.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440a83d0 con 0x7fb044141910 2023-10-30T20:45:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440b4f80 con 0x7fb0440b4420 2023-10-30T20:45:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04413fe10 con 0x7fb0440b12d0 2023-10-30T20:45:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb0490cf640 10 monclient(hunting): authenticate will time out at 2882.772107s 2023-10-30T20:45:59.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.483+0000 7fb042d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0440b12d0 0x7fb0441da1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.484+0000 7fb04359e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb044141910 0x7fb0441d6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.484+0000 7fb042d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0440b12d0 0x7fb0441da1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:50282/0 (socket says 172.21.15.69:50282) 2023-10-30T20:45:59.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.484+0000 7fb042d9d640 1 -- 172.21.15.69:0/2292048033 learned_addr learned my addr 172.21.15.69:0/2292048033 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:59.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.484+0000 7fb042d9d640 10 monclient(hunting): get_auth_request con 0x7fb0440b12d0 auth_method 0 2023-10-30T20:45:59.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.484+0000 7fb042d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:59.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.484+0000 7fb042d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:59.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.484+0000 7fb042d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:59.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.484+0000 7fb04359e640 10 monclient(hunting): get_auth_request con 0x7fb044141910 auth_method 0 2023-10-30T20:45:59.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.484+0000 7fb04359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:59.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.484+0000 7fb04359e640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:59.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.484+0000 7fb04359e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:59.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.484+0000 7fb042d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:59.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.484+0000 7fb042d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:59.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.484+0000 7fb042d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:59.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.485+0000 7fb04359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.485+0000 7fb04359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.485+0000 7fb04359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.487+0000 7fb04359e640 10 monclient(hunting): handle_auth_done global_id 16317 payload 824 2023-10-30T20:45:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.487+0000 7fb04359e640 1 -- 172.21.15.69:0/2292048033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0440b12d0 msgr2=0x7fb0441da1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.487+0000 7fb04359e640 1 --2- 172.21.15.69:0/2292048033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0440b12d0 0x7fb0441da1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.487+0000 7fb04359e640 1 -- 172.21.15.69:0/2292048033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0440b4420 msgr2=0x7fb0441da6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.487+0000 7fb04359e640 1 --2- 172.21.15.69:0/2292048033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0440b4420 0x7fb0441da6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.487+0000 7fb04359e640 10 monclient: _finish_hunting 0 2023-10-30T20:45:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.487+0000 7fb04359e640 1 monclient: found mon.noname-a 2023-10-30T20:45:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.487+0000 7fb04359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.487+0000 7fb04359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:45:59.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.487+0000 7fb04359e640 1 -- 172.21.15.69:0/2292048033 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb044140a50 con 0x7fb044141910 2023-10-30T20:45:59.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.490+0000 7fb04359e640 1 --2- 172.21.15.69:0/2292048033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb044141910 0x7fb0441d6a30 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 crypto rx=0x7fb03c033000 tx=0x7fb03c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:59.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.490+0000 7fb03b7fe640 10 client.?.objecter ms_handle_connect 0x7fb044141910 2023-10-30T20:45:59.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.490+0000 7fb03b7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:59.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb03b7fe640 1 -- 172.21.15.69:0/2292048033 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c040650 con 0x7fb044141910 2023-10-30T20:45:59.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb03b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:59.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb03b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:45:59.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb03b7fe640 10 monclient: dump: 2023-10-30T20:45:59.522 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:59.522 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:59.523 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.523 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.523 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:59.523 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:59.523 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:59.524 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:59.524 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:59.524 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:59.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb03b7fe640 10 monclient: _finish_auth 0 2023-10-30T20:45:59.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb03b7fe640 10 monclient: _check_auth_tickets 2023-10-30T20:45:59.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb03b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:59.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb0490cf640 5 monclient: authenticate success, global_id 16317 2023-10-30T20:45:59.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb0490cf640 10 monclient: _renew_subs 2023-10-30T20:45:59.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb0490cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:45:59.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb0490cf640 1 -- 172.21.15.69:0/2292048033 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0441dac20 con 0x7fb044141910 2023-10-30T20:45:59.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb0490cf640 10 client.16317.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:59.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb0490cf640 10 monclient: _renew_subs 2023-10-30T20:45:59.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb0490cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:45:59.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb0490cf640 1 -- 172.21.15.69:0/2292048033 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0440b6da0 con 0x7fb044141910 2023-10-30T20:45:59.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.493+0000 7fb0490cf640 1 librados: init done 2023-10-30T20:45:59.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.494+0000 7fb0490cf640 10 client.16317.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:59.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.494+0000 7fb0490cf640 15 client.16317.objecter _session_command_op_assign -1 1 2023-10-30T20:45:59.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.494+0000 7fb0490cf640 20 client.16317.objecter _get_session osd=-1 returning homeless 2023-10-30T20:45:59.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.494+0000 7fb0490cf640 10 client.16317.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:59.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.494+0000 7fb0490cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:45:59.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.494+0000 7fb0490cf640 1 -- 172.21.15.69:0/2292048033 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb0440aa740 con 0x7fb044141910 2023-10-30T20:45:59.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.495+0000 7fb03b7fe640 1 -- 172.21.15.69:0/2292048033 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0440aa740 con 0x7fb044141910 2023-10-30T20:45:59.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.495+0000 7fb03b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:59.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.495+0000 7fb03b7fe640 1 -- 172.21.15.69:0/2292048033 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c009a70 con 0x7fb044141910 2023-10-30T20:45:59.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.495+0000 7fb03b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:59.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.495+0000 7fb03b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:45:59.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.495+0000 7fb03b7fe640 10 monclient: dump: 2023-10-30T20:45:59.530 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:59.530 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:59.530 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.531 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.531 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:59.531 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:59.531 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:59.531 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:59.532 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:59.532 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:59.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.495+0000 7fb041d9b640 4 set_mon_vals no callback set 2023-10-30T20:45:59.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.517+0000 7fb03b7fe640 1 -- 172.21.15.69:0/2292048033 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fb03c0407c0 con 0x7fb044141910 2023-10-30T20:45:59.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.517+0000 7fb03b7fe640 1 --2- 172.21.15.69:0/2292048033 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb02406c7c0 0x7fb02406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.517+0000 7fb042d9d640 1 --2- 172.21.15.69:0/2292048033 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb02406c7c0 0x7fb02406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.518+0000 7fb042d9d640 10 monclient: get_auth_request con 0x7fb02406c7c0 auth_method 0 2023-10-30T20:45:59.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.518+0000 7fb042d9d640 1 --2- 172.21.15.69:0/2292048033 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb02406c7c0 0x7fb02406ec80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb0340094a0 tx=0x7fb034009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:59.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.518+0000 7fb03b7fe640 10 client.16317.objecter ms_handle_connect 0x7fb02406c7c0 2023-10-30T20:45:59.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 1 -- 172.21.15.69:0/2292048033 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7fb03c02ce30 con 0x7fb044141910 2023-10-30T20:45:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 10 client.16317.objecter ms_dispatch 0x7fb044000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:45:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 3 client.16317.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:45:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 3 client.16317.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:45:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 10 client.16317.objecter checking command 1 2023-10-30T20:45:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 1 --2- 172.21.15.69:0/2292048033 >> v2:172.21.15.183:6804/1046221745 conn(0x7fb024073c00 0x7fb0240760a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 20 client.16317.objecter _get_session s=0x7fb03c009a70 osd=6 3 2023-10-30T20:45:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 20 client.16317.objecter put_session s=0x7fb03c009a70 osd=6 3 2023-10-30T20:45:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 15 client.16317.objecter _session_command_op_remove -1 1 2023-10-30T20:45:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 20 client.16317.objecter _get_session s=0x7fb03c009a70 osd=6 3 2023-10-30T20:45:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 20 client.16317.objecter get_session s=0x7fb03c009a70 osd=6 3 2023-10-30T20:45:59.538 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 15 client.16317.objecter _session_command_op_assign 6 1 2023-10-30T20:45:59.538 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 20 client.16317.objecter put_session s=0x7fb03c009a70 osd=6 4 2023-10-30T20:45:59.538 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 10 client.16317.objecter _send_command 1 2023-10-30T20:45:59.538 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.520+0000 7fb03b7fe640 1 -- 172.21.15.69:0/2292048033 --> v2:172.21.15.183:6804/1046221745 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb03c040650 con 0x7fb024073c00 2023-10-30T20:45:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.521+0000 7fb03b7fe640 20 client.16317.objecter dump_active .. 0 homeless 2023-10-30T20:45:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.521+0000 7fb03b7fe640 1 -- 172.21.15.69:0/2292048033 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=904) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb03c076730 con 0x7fb044141910 2023-10-30T20:45:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.521+0000 7fb03b7fe640 10 monclient: handle_get_version_reply finishing 1 version 904 2023-10-30T20:45:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.521+0000 7fb03bfff640 1 --2- 172.21.15.69:0/2292048033 >> v2:172.21.15.183:6804/1046221745 conn(0x7fb024073c00 0x7fb0240760a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.522+0000 7fb03bfff640 10 monclient: get_auth_request con 0x7fb024073c00 auth_method 0 2023-10-30T20:45:59.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.524+0000 7fb03bfff640 1 --2- 172.21.15.69:0/2292048033 >> v2:172.21.15.183:6804/1046221745 conn(0x7fb024073c00 0x7fb0240760a0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:59.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.524+0000 7fb03b7fe640 10 client.16317.objecter ms_handle_connect 0x7fb024073c00 2023-10-30T20:45:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.546+0000 7fb03b7fe640 1 -- 172.21.15.69:0/2292048033 <== osd.6 v2:172.21.15.183:6804/1046221745 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fb03c040650 con 0x7fb024073c00 2023-10-30T20:45:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.546+0000 7fb03b7fe640 10 client.16317.objecter ms_dispatch 0x7fb044000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:45:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.546+0000 7fb03b7fe640 10 client.16317.objecter _finish_command 1 = system:0 2023-10-30T20:45:59.551 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.546+0000 7fb03b7fe640 20 client.16317.objecter put_session s=0x7fb03c009a70 osd=6 4 2023-10-30T20:45:59.551 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.546+0000 7fb03b7fe640 15 client.16317.objecter _session_command_op_remove 6 1 2023-10-30T20:45:59.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.561+0000 7fb0490cf640 10 client.16317.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T20:45:59.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.561+0000 7fb0490cf640 15 client.16317.objecter _session_command_op_assign -1 2 2023-10-30T20:45:59.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.561+0000 7fb0490cf640 20 client.16317.objecter _get_session s=0x7fb03c009a70 osd=6 3 2023-10-30T20:45:59.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.561+0000 7fb0490cf640 20 client.16317.objecter put_session s=0x7fb03c009a70 osd=6 3 2023-10-30T20:45:59.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.561+0000 7fb0490cf640 20 client.16317.objecter _get_session s=0x7fb03c009a70 osd=6 3 2023-10-30T20:45:59.564 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.561+0000 7fb0490cf640 15 client.16317.objecter _session_command_op_remove -1 2 2023-10-30T20:45:59.564 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.561+0000 7fb0490cf640 20 client.16317.objecter get_session s=0x7fb03c009a70 osd=6 3 2023-10-30T20:45:59.564 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.561+0000 7fb0490cf640 15 client.16317.objecter _session_command_op_assign 6 2 2023-10-30T20:45:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.561+0000 7fb0490cf640 20 client.16317.objecter put_session s=0x7fb03c009a70 osd=6 4 2023-10-30T20:45:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.561+0000 7fb0490cf640 10 client.16317.objecter _send_command 2 2023-10-30T20:45:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.561+0000 7fb0490cf640 1 -- 172.21.15.69:0/2292048033 --> v2:172.21.15.183:6804/1046221745 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb0440a87e0 con 0x7fb024073c00 2023-10-30T20:45:59.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb03b7fe640 1 -- 172.21.15.69:0/2292048033 <== osd.6 v2:172.21.15.183:6804/1046221745 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fb02c0096c0 con 0x7fb024073c00 2023-10-30T20:45:59.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb03b7fe640 10 client.16317.objecter ms_dispatch 0x7fb044000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:45:59.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb03b7fe640 10 client.16317.objecter _finish_command 2 = system:0 2023-10-30T20:45:59.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb03b7fe640 20 client.16317.objecter put_session s=0x7fb03c009a70 osd=6 4 2023-10-30T20:45:59.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb03b7fe640 15 client.16317.objecter _session_command_op_remove 6 2 2023-10-30T20:45:59.578 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-10-30T20:45:59.578 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T20:45:59.578 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-10-30T20:45:59.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 10 librados: watch_flush enter 2023-10-30T20:45:59.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 10 librados: watch_flush exit 2023-10-30T20:45:59.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 10 client.16317.objecter close_session for osd.6 2023-10-30T20:45:59.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 1 -- 172.21.15.69:0/2292048033 >> v2:172.21.15.183:6804/1046221745 conn(0x7fb024073c00 msgr2=0x7fb0240760a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:59.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 1 --2- 172.21.15.69:0/2292048033 >> v2:172.21.15.183:6804/1046221745 conn(0x7fb024073c00 0x7fb0240760a0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 20 client.16317.objecter put_session s=0x7fb03c009a70 osd=6 1 2023-10-30T20:45:59.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 20 client.16317.objecter shutdown clearing up homeless session... 2023-10-30T20:45:59.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 10 client.16317.objecter successfully canceled tick 2023-10-30T20:45:59.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 1 -- 172.21.15.69:0/2292048033 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb02406c7c0 msgr2=0x7fb02406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:59.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 1 --2- 172.21.15.69:0/2292048033 >> v2:172.21.15.69:6800/1876307353 conn(0x7fb02406c7c0 0x7fb02406ec80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb0340094a0 tx=0x7fb034009240 comp rx=0 tx=0).stop 2023-10-30T20:45:59.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 10 monclient: shutdown 2023-10-30T20:45:59.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:59.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 1 -- 172.21.15.69:0/2292048033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb044141910 msgr2=0x7fb0441d6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:59.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 1 --2- 172.21.15.69:0/2292048033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb044141910 0x7fb0441d6a30 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 crypto rx=0x7fb03c033000 tx=0x7fb03c004a40 comp rx=0 tx=0).stop 2023-10-30T20:45:59.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb04359e640 1 -- 172.21.15.69:0/2292048033 reap_dead start 2023-10-30T20:45:59.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 1 -- 172.21.15.69:0/2292048033 shutdown_connections 2023-10-30T20:45:59.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 1 -- 172.21.15.69:0/2292048033 >> 172.21.15.69:0/2292048033 conn(0x7fb044065ea0 msgr2=0x7fb04413ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:59.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 1 -- 172.21.15.69:0/2292048033 shutdown_connections 2023-10-30T20:45:59.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 1 -- 172.21.15.69:0/2292048033 wait complete. 2023-10-30T20:45:59.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.576+0000 7fb0490cf640 1 librados: shutdown 2023-10-30T20:45:59.588 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-30T20:45:59.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.696+0000 7f674600e640 10 monclient: get_monmap_and_config 2023-10-30T20:45:59.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.696+0000 7f674600e640 10 monclient: build_initial_monmap 2023-10-30T20:45:59.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.696+0000 7f674600e640 10 monclient: monmap: 2023-10-30T20:45:59.698 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:59.698 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:59.698 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:45:59.697133+0000 2023-10-30T20:45:59.698 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:45:59.697133+0000 2023-10-30T20:45:59.698 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:45:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:45:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:45:59.700 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:45:59.700 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:45:59.700 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:59.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.696+0000 7f674600e640 1 Processor -- start 2023-10-30T20:45:59.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.696+0000 7f674600e640 1 -- start start 2023-10-30T20:45:59.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.696+0000 7f674600e640 10 monclient: init 2023-10-30T20:45:59.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.697+0000 7f674600e640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:59.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.697+0000 7f674600e640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-10-30T20:45:59.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.697+0000 7f674600e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67400b4350 addr v1:172.21.15.183:6789/0 2023-10-30T20:45:59.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.697+0000 7f674600e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f674013eb50 addr v1:172.21.15.69:6789/0 2023-10-30T20:45:59.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.697+0000 7f674600e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6740144f50 0x7f67401453a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.697+0000 7f674600e640 10 monclient(hunting): picked mon.noname-c con 0x7f6740144f50 addr v2:172.21.15.183:3300/0 2023-10-30T20:45:59.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.697+0000 7f674600e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674013f6b0 con 0x7f674013eb50 2023-10-30T20:45:59.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.697+0000 7f674600e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67400aa6e0 con 0x7f67400b4350 2023-10-30T20:45:59.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.697+0000 7f674600e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:59.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.697+0000 7f674600e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67400679f0 con 0x7f6740144f50 2023-10-30T20:45:59.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.697+0000 7f674600e640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:59.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.697+0000 7f674600e640 10 monclient(hunting): authenticate will time out at 2882.986415s 2023-10-30T20:45:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.727+0000 7f673ffff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6740144f50 0x7f67401453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.728+0000 7f673ffff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6740144f50 0x7f67401453a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:54118/0 (socket says 172.21.15.69:54118) 2023-10-30T20:45:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.728+0000 7f673ffff640 1 -- 172.21.15.69:0/2338433373 learned_addr learned my addr 172.21.15.69:0/2338433373 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:59.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.728+0000 7f673ffff640 10 monclient(hunting): get_auth_request con 0x7f6740144f50 auth_method 0 2023-10-30T20:45:59.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.728+0000 7f673ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:59.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.728+0000 7f673ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:59.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.728+0000 7f673ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:59.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.729+0000 7f673ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:59.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.729+0000 7f673ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:59.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.729+0000 7f673ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:59.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.731+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765584265 0 0) 0x7f674013f6b0 con 0x7f674013eb50 2023-10-30T20:45:59.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.731+0000 7f673e7fc640 10 monclient(hunting): _negotiate 2023-10-30T20:45:59.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.731+0000 7f673e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:59.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.731+0000 7f673e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:59.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.731+0000 7f673e7fc640 10 monclient(hunting): my global_id is 16320 2023-10-30T20:45:59.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.731+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6728003520 con 0x7f674013eb50 2023-10-30T20:45:59.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.733+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788461743 0 0) 0x7f6730001110 con 0x7f67400b4350 2023-10-30T20:45:59.735 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.733+0000 7f673e7fc640 10 monclient(hunting): _negotiate 2023-10-30T20:45:59.735 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.733+0000 7f673e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:59.735 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.733+0000 7f673e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:59.735 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.733+0000 7f673e7fc640 10 monclient(hunting): my global_id is 21298 2023-10-30T20:45:59.735 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.733+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6728003840 con 0x7f67400b4350 2023-10-30T20:45:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 465220412 0 0) 0x7f6734001370 con 0x7f674013eb50 2023-10-30T20:45:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 >> v2:172.21.15.183:3300/0 conn(0x7f6740144f50 msgr2=0x7f67401453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 1 --2- 172.21.15.69:0/2338433373 >> v2:172.21.15.183:3300/0 conn(0x7f6740144f50 0x7f67401453a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 >> v1:172.21.15.183:6789/0 conn(0x7f67400b4350 legacy=0x7f67400a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 10 monclient: _finish_hunting 0 2023-10-30T20:45:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 1 monclient: found mon.noname-a-legacy 2023-10-30T20:45:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.69:6789/0 2023-10-30T20:45:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67400a8bc0 con 0x7f674013eb50 2023-10-30T20:45:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 10 monclient: _finish_auth 0 2023-10-30T20:45:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 10 monclient: _check_auth_tickets 2023-10-30T20:45:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 396370546 0 0) 0x7f6734001810 con 0x7f674013eb50 2023-10-30T20:45:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T20:45:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 10 monclient: dump: 2023-10-30T20:45:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.742 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.742 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:59.742 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:59.742 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:59.742 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:59.743 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:59.743 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:59.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] but i'm connected to v1:172.21.15.69:6789/0, reconnecting 2023-10-30T20:45:59.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:59.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 >> v1:172.21.15.69:6789/0 conn(0x7f674013eb50 legacy=0x7f674013efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:59.744 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T20:45:59.744 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 1 --2- 172.21.15.69:0/2338433373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6728003ce0 0x7f67280040e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.744 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.734+0000 7f673e7fc640 10 monclient(hunting): picked mon.c con 0x7f6728003ce0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:59.744 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.735+0000 7f673e7fc640 1 --2- 172.21.15.69:0/2338433373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6728004880 0x7f6728004d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.744 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.735+0000 7f673e7fc640 10 monclient(hunting): picked mon.a con 0x7f6728004880 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:59.745 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.735+0000 7f673e7fc640 1 --2- 172.21.15.69:0/2338433373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6728009970 0x7f672800bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.745 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.735+0000 7f673e7fc640 10 monclient(hunting): picked mon.b con 0x7f6728009970 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:59.745 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.735+0000 7f673e7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:59.745 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.735+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67400679f0 con 0x7f6728004880 2023-10-30T20:45:59.745 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.735+0000 7f673e7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:59.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.735+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672800c320 con 0x7f6728009970 2023-10-30T20:45:59.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.735+0000 7f673e7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:59.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.735+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672800c4f0 con 0x7f6728003ce0 2023-10-30T20:45:59.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.735+0000 7f673e7fc640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:59.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.735+0000 7f673e7fc640 10 monclient(hunting): _finish_auth 0 2023-10-30T20:45:59.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.737+0000 7f673ffff640 1 --2- 172.21.15.69:0/2338433373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6728009970 0x7f672800bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.737+0000 7f673ffff640 10 monclient(hunting): get_auth_request con 0x7f6728009970 auth_method 0 2023-10-30T20:45:59.750 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.737+0000 7f673ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:59.750 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.737+0000 7f673ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:59.750 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.737+0000 7f673ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:45:59.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.738+0000 7f673f7fe640 1 --2- 172.21.15.69:0/2338433373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6728003ce0 0x7f67280040e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.738+0000 7f673effd640 1 --2- 172.21.15.69:0/2338433373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6728004880 0x7f6728004d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.739+0000 7f673ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:59.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.739+0000 7f673ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:59.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.739+0000 7f673ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T20:45:59.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.739+0000 7f673f7fe640 10 monclient(hunting): get_auth_request con 0x7f6728003ce0 auth_method 0 2023-10-30T20:45:59.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.739+0000 7f673f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:59.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.739+0000 7f673f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:59.753 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.739+0000 7f673f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:45:59.753 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.739+0000 7f673effd640 10 monclient(hunting): get_auth_request con 0x7f6728004880 auth_method 0 2023-10-30T20:45:59.753 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.739+0000 7f673effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:59.753 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.739+0000 7f673effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:59.753 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.739+0000 7f673effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T20:45:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T20:45:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T20:45:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673ffff640 10 monclient(hunting): handle_auth_done global_id 16320 payload 293 2023-10-30T20:45:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673ffff640 1 -- 172.21.15.69:0/2338433373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6728003ce0 msgr2=0x7f67280040e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673ffff640 1 --2- 172.21.15.69:0/2338433373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6728003ce0 0x7f67280040e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673ffff640 1 -- 172.21.15.69:0/2338433373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6728004880 msgr2=0x7f6728004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:45:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673ffff640 1 --2- 172.21.15.69:0/2338433373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6728004880 0x7f6728004d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673ffff640 10 monclient: _finish_hunting 0 2023-10-30T20:45:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673ffff640 1 monclient: found mon.b 2023-10-30T20:45:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673ffff640 1 -- 172.21.15.69:0/2338433373 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6734001370 con 0x7f6728009970 2023-10-30T20:45:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673ffff640 10 monclient: _finish_auth 0 2023-10-30T20:45:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673ffff640 10 monclient: _check_auth_tickets 2023-10-30T20:45:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f673ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f674600e640 5 monclient: authenticate success, global_id 16320 2023-10-30T20:45:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.740+0000 7f674600e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:45:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.741+0000 7f673ffff640 1 -- 172.21.15.69:0/2338433373 reap_dead start 2023-10-30T20:45:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.743+0000 7f673ffff640 1 --2- 172.21.15.69:0/2338433373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6728009970 0x7f672800bd60 secure :-1 s=READY pgs=5067 cs=0 l=1 rev1=1 crypto rx=0x7f673800e1c0 tx=0x7f67280057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfb5fb59ca3d2bfa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.747+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673800f040 con 0x7f6728009970 2023-10-30T20:45:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.747+0000 7f673e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.747+0000 7f673e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.747+0000 7f673e7fc640 10 monclient: dump: 2023-10-30T20:45:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:59.761 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:59.761 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:59.761 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:59.761 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:59.761 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:59.762 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.747+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673802cae0 con 0x7f6728009970 2023-10-30T20:45:59.762 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.747+0000 7f673e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:59.762 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.747+0000 7f673e7fc640 1 -- 172.21.15.69:0/2338433373 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673802ce80 con 0x7f6728009970 2023-10-30T20:45:59.762 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.747+0000 7f674600e640 10 monclient: get_monmap_and_config success 2023-10-30T20:45:59.762 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.747+0000 7f674600e640 4 set_mon_vals no callback set 2023-10-30T20:45:59.763 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.747+0000 7f673e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:59.763 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.747+0000 7f673e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:59.763 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.747+0000 7f673e7fc640 10 monclient: dump: 2023-10-30T20:45:59.763 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:59.763 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:59.764 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.764 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.764 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:59.764 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:59.764 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:59.765 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:59.765 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:59.765 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:59.765 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.748+0000 7f674600e640 10 monclient: shutdown 2023-10-30T20:45:59.765 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.748+0000 7f674600e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:45:59.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.748+0000 7f674600e640 1 -- 172.21.15.69:0/2338433373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6728009970 msgr2=0x7f672800bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:59.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.748+0000 7f674600e640 1 --2- 172.21.15.69:0/2338433373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6728009970 0x7f672800bd60 secure :-1 s=READY pgs=5067 cs=0 l=1 rev1=1 crypto rx=0x7f673800e1c0 tx=0x7f67280057f0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.772 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.748+0000 7f674600e640 1 -- 172.21.15.69:0/2338433373 shutdown_connections 2023-10-30T20:45:59.772 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.748+0000 7f674600e640 1 --2- 172.21.15.69:0/2338433373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6728009970 0x7f672800bd60 unknown :-1 s=CLOSED pgs=5067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.772 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.748+0000 7f674600e640 1 -- 172.21.15.69:0/2338433373 >> 172.21.15.69:0/2338433373 conn(0x7f6740065ea0 msgr2=0x7f67400662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:45:59.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.748+0000 7f674600e640 1 -- 172.21.15.69:0/2338433373 shutdown_connections 2023-10-30T20:45:59.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.748+0000 7f674600e640 1 -- 172.21.15.69:0/2338433373 wait complete. 2023-10-30T20:45:59.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.748+0000 7f674600e640 10 monclient: build_initial_monmap 2023-10-30T20:45:59.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.748+0000 7f674600e640 10 monclient: monmap: 2023-10-30T20:45:59.773 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:45:59.774 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:45:59.774 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:45:59.774 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:45:59.774 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:45:59.774 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:45:59.775 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:45:59.775 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:45:59.775 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:45:59.775 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:59.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.749+0000 7f674600e640 1 librados: starting msgr at 2023-10-30T20:45:59.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.749+0000 7f674600e640 1 librados: starting objecter 2023-10-30T20:45:59.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.749+0000 7f674600e640 1 Processor -- start 2023-10-30T20:45:59.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.749+0000 7f674600e640 1 -- start start 2023-10-30T20:45:59.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.749+0000 7f674600e640 1 librados: setting wanted keys 2023-10-30T20:45:59.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.749+0000 7f674600e640 1 librados: calling monclient init 2023-10-30T20:45:59.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.749+0000 7f674600e640 10 monclient: init 2023-10-30T20:45:59.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 10 monclient: _reopen_session rank -1 2023-10-30T20:45:59.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:45:59.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6728009970 0x7f67400ac690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 10 monclient(hunting): picked mon.noname-b con 0x7f6728009970 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:45:59.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67401cbd20 0x7f6740138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 10 monclient(hunting): picked mon.noname-c con 0x7f67401cbd20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:45:59.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6740138a40 0x7f67401d69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 10 monclient(hunting): picked mon.noname-a con 0x7f6740138a40 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:45:59.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:59.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67400a9050 con 0x7f6740138a40 2023-10-30T20:45:59.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:59.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67401cc480 con 0x7f6728009970 2023-10-30T20:45:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:45:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67401cc180 con 0x7f67401cbd20 2023-10-30T20:45:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 10 monclient(hunting): _renew_subs 2023-10-30T20:45:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f674600e640 10 monclient(hunting): authenticate will time out at 2883.038982s 2023-10-30T20:45:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f673effd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67401cbd20 0x7f6740138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f673ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6740138a40 0x7f67401d69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.750+0000 7f673f7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6728009970 0x7f67400ac690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6740138a40 0x7f67401d69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50300/0 (socket says 172.21.15.69:50300) 2023-10-30T20:45:59.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673f7fe640 10 monclient(hunting): get_auth_request con 0x7f6728009970 auth_method 0 2023-10-30T20:45:59.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673ffff640 1 -- 172.21.15.69:0/4135247714 learned_addr learned my addr 172.21.15.69:0/4135247714 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:45:59.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:59.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:59.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:59.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673ffff640 10 monclient(hunting): get_auth_request con 0x7f6740138a40 auth_method 0 2023-10-30T20:45:59.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:59.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:59.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:59.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673effd640 10 monclient(hunting): get_auth_request con 0x7f67401cbd20 auth_method 0 2023-10-30T20:45:59.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:45:59.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:45:59.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:45:59.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:59.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:59.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:59.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:59.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:59.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:59.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:45:59.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:45:59.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.751+0000 7f673effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:45:59.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.754+0000 7f673f7fe640 10 monclient(hunting): handle_auth_done global_id 21301 payload 824 2023-10-30T20:45:59.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.754+0000 7f673f7fe640 1 -- 172.21.15.69:0/4135247714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67401cbd20 msgr2=0x7f6740138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:59.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.754+0000 7f673f7fe640 1 --2- 172.21.15.69:0/4135247714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67401cbd20 0x7f6740138500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.754+0000 7f673f7fe640 1 -- 172.21.15.69:0/4135247714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6740138a40 msgr2=0x7f67401d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:45:59.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.754+0000 7f673f7fe640 1 --2- 172.21.15.69:0/4135247714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6740138a40 0x7f67401d69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:45:59.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.754+0000 7f673f7fe640 10 monclient: _finish_hunting 0 2023-10-30T20:45:59.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.754+0000 7f673f7fe640 1 monclient: found mon.noname-b 2023-10-30T20:45:59.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.754+0000 7f673f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:45:59.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.754+0000 7f673f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:45:59.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.754+0000 7f673f7fe640 1 -- 172.21.15.69:0/4135247714 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67401cb1e0 con 0x7f6728009970 2023-10-30T20:45:59.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.754+0000 7f673ffff640 1 --2- 172.21.15.69:0/4135247714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6740138a40 0x7f67401d69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T20:45:59.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.754+0000 7f673effd640 1 --2- 172.21.15.69:0/4135247714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67401cbd20 0x7f6740138500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T20:45:59.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.759+0000 7f673f7fe640 1 --2- 172.21.15.69:0/4135247714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6728009970 0x7f67400ac690 secure :-1 s=READY pgs=5068 cs=0 l=1 rev1=1 crypto rx=0x7f673000f7d0 tx=0x7f673000fc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:45:59.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.759+0000 7f6727fff640 10 client.?.objecter ms_handle_connect 0x7f6728009970 2023-10-30T20:45:59.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.759+0000 7f6727fff640 10 client.?.objecter resend_mon_ops 2023-10-30T20:45:59.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f6727fff640 1 -- 172.21.15.69:0/4135247714 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6730005110 con 0x7f6728009970 2023-10-30T20:45:59.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f6727fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:59.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f6727fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:45:59.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f6727fff640 10 monclient: dump: 2023-10-30T20:45:59.790 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:59.790 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:59.791 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.791 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.791 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:59.791 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:59.791 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:59.792 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:59.792 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:59.792 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:59.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f6727fff640 10 monclient: _finish_auth 0 2023-10-30T20:45:59.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f6727fff640 10 monclient: _check_auth_tickets 2023-10-30T20:45:59.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f6727fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:45:59.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f6727fff640 1 -- 172.21.15.69:0/4135247714 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673000e040 con 0x7f6728009970 2023-10-30T20:45:59.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f6727fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:45:59.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f6727fff640 1 -- 172.21.15.69:0/4135247714 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6730007050 con 0x7f6728009970 2023-10-30T20:45:59.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f674600e640 5 monclient: authenticate success, global_id 21301 2023-10-30T20:45:59.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f6727fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:45:59.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f6727fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:45:59.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f6727fff640 10 monclient: dump: 2023-10-30T20:45:59.794 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:45:59.794 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:45:59.795 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.795 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:45:59.795 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:45:59.795 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:45:59.795 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:45:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:45:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:45:59.796 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:45:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f674600e640 10 monclient: _renew_subs 2023-10-30T20:45:59.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f674600e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:59.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f673dffb640 4 set_mon_vals no callback set 2023-10-30T20:45:59.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f674600e640 1 -- 172.21.15.69:0/4135247714 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67400ad200 con 0x7f6728009970 2023-10-30T20:45:59.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f674600e640 10 client.21301.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:45:59.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f674600e640 10 monclient: _renew_subs 2023-10-30T20:45:59.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f674600e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:59.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.764+0000 7f674600e640 1 -- 172.21.15.69:0/4135247714 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67401d70c0 con 0x7f6728009970 2023-10-30T20:45:59.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.765+0000 7f674600e640 1 librados: init done 2023-10-30T20:45:59.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.765+0000 7f674600e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:59.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.765+0000 7f674600e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:45:59.799 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.765+0000 7f674600e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:45:59.799 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.765+0000 7f674600e640 1 -- 172.21.15.69:0/4135247714 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67401cbb20 con 0x7f6728009970 2023-10-30T20:45:59.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.785+0000 7f6727fff640 1 -- 172.21.15.69:0/4135247714 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f67300071c0 con 0x7f6728009970 2023-10-30T20:45:59.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.785+0000 7f6727fff640 1 --2- 172.21.15.69:0/4135247714 >> v2:172.21.15.69:6800/1876307353 conn(0x7f672c06c840 0x7f672c06ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:45:59.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.785+0000 7f673effd640 1 --2- 172.21.15.69:0/4135247714 >> v2:172.21.15.69:6800/1876307353 conn(0x7f672c06c840 0x7f672c06ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:45:59.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.785+0000 7f673effd640 10 monclient: get_auth_request con 0x7f672c06c840 auth_method 0 2023-10-30T20:45:59.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.785+0000 7f6727fff640 10 client.21301.objecter ms_handle_connect 0x7f672c06c840 2023-10-30T20:45:59.801 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:45:59.785+0000 7f673effd640 1 --2- 172.21.15.69:0/4135247714 >> v2:172.21.15.69:6800/1876307353 conn(0x7f672c06c840 0x7f672c06ed00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6734007fd0 tx=0x7f6734009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:00.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.018+0000 7f6727fff640 1 -- 172.21.15.69:0/4135247714 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f6730057b10 con 0x7f6728009970 2023-10-30T20:46:00.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.018+0000 7f6727fff640 10 client.21301.objecter ms_dispatch 0x7f6740000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:46:00.020 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.018+0000 7f6727fff640 3 client.21301.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:46:00.020 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.018+0000 7f6727fff640 3 client.21301.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:46:00.020 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.018+0000 7f6727fff640 20 client.21301.objecter dump_active .. 0 homeless 2023-10-30T20:46:00.030 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.029+0000 7f6727fff640 1 -- 172.21.15.69:0/4135247714 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f673005bbb0 con 0x7f6728009970 2023-10-30T20:46:00.030 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.029+0000 7f6727fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:00.030 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.029+0000 7f6727fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:46:00.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.199+0000 7f674600e640 1 -- 172.21.15.69:0/4135247714 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f67400adf60 con 0x7f672c06c840 2023-10-30T20:46:00.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.199+0000 7f6727fff640 1 -- 172.21.15.69:0/4135247714 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6734012de0 con 0x7f672c06c840 2023-10-30T20:46:00.201 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 6 to deep-scrub 2023-10-30T20:46:00.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 10 librados: watch_flush enter 2023-10-30T20:46:00.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 10 librados: watch_flush exit 2023-10-30T20:46:00.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 20 client.21301.objecter shutdown clearing up homeless session... 2023-10-30T20:46:00.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 10 client.21301.objecter successfully canceled tick 2023-10-30T20:46:00.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 1 -- 172.21.15.69:0/4135247714 >> v2:172.21.15.69:6800/1876307353 conn(0x7f672c06c840 msgr2=0x7f672c06ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:00.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 1 --2- 172.21.15.69:0/4135247714 >> v2:172.21.15.69:6800/1876307353 conn(0x7f672c06c840 0x7f672c06ed00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6734007fd0 tx=0x7f6734009040 comp rx=0 tx=0).stop 2023-10-30T20:46:00.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 10 monclient: shutdown 2023-10-30T20:46:00.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:46:00.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 1 -- 172.21.15.69:0/4135247714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6728009970 msgr2=0x7f67400ac690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:00.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 1 --2- 172.21.15.69:0/4135247714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6728009970 0x7f67400ac690 secure :-1 s=READY pgs=5068 cs=0 l=1 rev1=1 crypto rx=0x7f673000f7d0 tx=0x7f673000fc90 comp rx=0 tx=0).stop 2023-10-30T20:46:00.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 1 -- 172.21.15.69:0/4135247714 shutdown_connections 2023-10-30T20:46:00.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 1 --2- 172.21.15.69:0/4135247714 >> v2:172.21.15.69:6800/1876307353 conn(0x7f672c06c840 0x7f672c06ed00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 1 --2- 172.21.15.69:0/4135247714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6740138a40 0x7f67401d69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 1 --2- 172.21.15.69:0/4135247714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67401cbd20 0x7f6740138500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 1 --2- 172.21.15.69:0/4135247714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6728009970 0x7f67400ac690 unknown :-1 s=CLOSED pgs=5068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 1 -- 172.21.15.69:0/4135247714 >> 172.21.15.69:0/4135247714 conn(0x7f6740065ea0 msgr2=0x7f674013cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:46:00.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 1 -- 172.21.15.69:0/4135247714 shutdown_connections 2023-10-30T20:46:00.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 1 -- 172.21.15.69:0/4135247714 wait complete. 2023-10-30T20:46:00.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.201+0000 7f674600e640 1 librados: shutdown 2023-10-30T20:46:00.213 INFO:tasks.ceph:Scrubbing osd.7 2023-10-30T20:46:00.214 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-30T20:46:00.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.323+0000 7f0ceea2d640 10 monclient: get_monmap_and_config 2023-10-30T20:46:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.323+0000 7f0ceea2d640 10 monclient: build_initial_monmap 2023-10-30T20:46:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.323+0000 7f0ceea2d640 10 monclient: monmap: 2023-10-30T20:46:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:46:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:46:00.324513+0000 2023-10-30T20:46:00.326 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:46:00.324513+0000 2023-10-30T20:46:00.326 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:46:00.326 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:00.326 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:46:00.326 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:46:00.327 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:46:00.327 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:46:00.327 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:46:00.327 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:00.328 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 1 Processor -- start 2023-10-30T20:46:00.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 1 -- start start 2023-10-30T20:46:00.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 10 monclient: init 2023-10-30T20:46:00.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 10 monclient: _reopen_session rank -1 2023-10-30T20:46:00.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-10-30T20:46:00.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ce80b4360 addr v1:172.21.15.69:6789/0 2023-10-30T20:46:00.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0ce813f320 0x7f0ce8144e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 10 monclient(hunting): picked mon.noname-a con 0x7f0ce813f320 addr v2:172.21.15.69:3300/0 2023-10-30T20:46:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce80b12b0 0x7f0ce8145350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 10 monclient(hunting): picked mon.noname-b con 0x7f0ce80b12b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:46:00.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce80aa180 con 0x7f0ce80b4360 2023-10-30T20:46:00.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:00.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce8067a30 con 0x7f0ce813f320 2023-10-30T20:46:00.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:00.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce80b4740 con 0x7f0ce80b12b0 2023-10-30T20:46:00.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 10 monclient(hunting): _renew_subs 2023-10-30T20:46:00.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.324+0000 7f0ceea2d640 10 monclient(hunting): authenticate will time out at 2883.613744s 2023-10-30T20:46:00.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.325+0000 7f0cecfca640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce80b12b0 0x7f0ce8145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:00.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.326+0000 7f0cecfca640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce80b12b0 0x7f0ce8145350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:50310/0 (socket says 172.21.15.69:50310) 2023-10-30T20:46:00.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.326+0000 7f0cecfca640 1 -- 172.21.15.69:0/708113678 learned_addr learned my addr 172.21.15.69:0/708113678 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:46:00.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.326+0000 7f0cecfca640 10 monclient(hunting): get_auth_request con 0x7f0ce80b12b0 auth_method 0 2023-10-30T20:46:00.333 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.326+0000 7f0cecfca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:00.333 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.326+0000 7f0cecfca640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:00.333 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.326+0000 7f0cecfca640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:00.333 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.326+0000 7f0cecfca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:00.333 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.326+0000 7f0cecfca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:00.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.326+0000 7f0cecfca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:00.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.328+0000 7f0cecfca640 10 monclient(hunting): handle_auth_done global_id 18341 payload 274 2023-10-30T20:46:00.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.328+0000 7f0cecfca640 1 -- 172.21.15.69:0/708113678 >> v2:172.21.15.69:3300/0 conn(0x7f0ce813f320 msgr2=0x7f0ce8144e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:46:00.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.328+0000 7f0cecfca640 1 --2- 172.21.15.69:0/708113678 >> v2:172.21.15.69:3300/0 conn(0x7f0ce813f320 0x7f0ce8144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.328+0000 7f0cecfca640 1 -- 172.21.15.69:0/708113678 >> v1:172.21.15.69:6789/0 conn(0x7f0ce80b4360 legacy=0x7f0ce813ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:46:00.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.328+0000 7f0cecfca640 10 monclient: _finish_hunting 0 2023-10-30T20:46:00.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.328+0000 7f0cecfca640 1 monclient: found mon.noname-b 2023-10-30T20:46:00.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.328+0000 7f0cecfca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:46:00.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.328+0000 7f0cecfca640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:46:00.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.328+0000 7f0cecfca640 1 -- 172.21.15.69:0/708113678 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce80aa3d0 con 0x7f0ce80b12b0 2023-10-30T20:46:00.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.330+0000 7f0cecfca640 1 --2- 172.21.15.69:0/708113678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce80b12b0 0x7f0ce8145350 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009e50 tx=0x7f0ce002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a93587b7cdeedc4c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:00.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.332+0000 7f0ce6ffd640 1 -- 172.21.15.69:0/708113678 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce002c9e0 con 0x7f0ce80b12b0 2023-10-30T20:46:00.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.332+0000 7f0ce6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:00.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.332+0000 7f0ce6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:46:00.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.332+0000 7f0ce6ffd640 10 monclient: dump: 2023-10-30T20:46:00.339 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:00.339 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:00.339 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.339 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.339 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:00.340 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:00.340 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:00.340 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:00.340 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:00.340 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:00.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.332+0000 7f0ce6ffd640 10 monclient: _finish_auth 0 2023-10-30T20:46:00.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.332+0000 7f0ce6ffd640 10 monclient: _check_auth_tickets 2023-10-30T20:46:00.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.332+0000 7f0ce6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:46:00.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.332+0000 7f0ceea2d640 5 monclient: authenticate success, global_id 18341 2023-10-30T20:46:00.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.332+0000 7f0ceea2d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:46:00.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.333+0000 7f0ce6ffd640 1 -- 172.21.15.69:0/708113678 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce002cb50 con 0x7f0ce80b12b0 2023-10-30T20:46:00.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.333+0000 7f0ce6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:46:00.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.333+0000 7f0ce6ffd640 1 -- 172.21.15.69:0/708113678 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce00343f0 con 0x7f0ce80b12b0 2023-10-30T20:46:00.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.333+0000 7f0ceea2d640 10 monclient: get_monmap_and_config success 2023-10-30T20:46:00.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.333+0000 7f0ceea2d640 4 set_mon_vals no callback set 2023-10-30T20:46:00.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.333+0000 7f0ce6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:00.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.333+0000 7f0ce6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:46:00.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.333+0000 7f0ce6ffd640 10 monclient: dump: 2023-10-30T20:46:00.345 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:00.346 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:00.346 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.347 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.347 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:00.348 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:00.348 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:00.349 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:00.349 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:00.350 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:00.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.333+0000 7f0ceea2d640 10 monclient: shutdown 2023-10-30T20:46:00.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.334+0000 7f0ceea2d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:46:00.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.334+0000 7f0ceea2d640 1 -- 172.21.15.69:0/708113678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce80b12b0 msgr2=0x7f0ce8145350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:00.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.334+0000 7f0ceea2d640 1 --2- 172.21.15.69:0/708113678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce80b12b0 0x7f0ce8145350 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009e50 tx=0x7f0ce002be40 comp rx=0 tx=0).stop 2023-10-30T20:46:00.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.334+0000 7f0ceea2d640 1 -- 172.21.15.69:0/708113678 shutdown_connections 2023-10-30T20:46:00.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.334+0000 7f0ceea2d640 1 --2- 172.21.15.69:0/708113678 >> v2:172.21.15.69:3300/0 conn(0x7f0ce813f320 0x7f0ce8144e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.334+0000 7f0ceea2d640 1 --2- 172.21.15.69:0/708113678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce80b12b0 0x7f0ce8145350 unknown :-1 s=CLOSED pgs=3211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.334+0000 7f0ceea2d640 1 -- 172.21.15.69:0/708113678 >> 172.21.15.69:0/708113678 conn(0x7f0ce8065ed0 msgr2=0x7f0ce80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:46:00.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.334+0000 7f0ceea2d640 1 -- 172.21.15.69:0/708113678 shutdown_connections 2023-10-30T20:46:00.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.334+0000 7f0ceea2d640 1 -- 172.21.15.69:0/708113678 wait complete. 2023-10-30T20:46:00.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.334+0000 7f0ceea2d640 10 monclient: build_initial_monmap 2023-10-30T20:46:00.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.334+0000 7f0ceea2d640 10 monclient: monmap: 2023-10-30T20:46:00.360 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:46:00.361 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:46:00.362 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:46:00.362 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:46:00.363 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:46:00.365 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:46:00.365 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:46:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:46:00.369 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:46:00.370 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:00.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.335+0000 7f0ceea2d640 1 librados: starting msgr at 2023-10-30T20:46:00.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.335+0000 7f0ceea2d640 1 librados: starting objecter 2023-10-30T20:46:00.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.335+0000 7f0ceea2d640 1 Processor -- start 2023-10-30T20:46:00.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.335+0000 7f0ceea2d640 1 -- start start 2023-10-30T20:46:00.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.335+0000 7f0ceea2d640 1 librados: setting wanted keys 2023-10-30T20:46:00.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.335+0000 7f0ceea2d640 1 librados: calling monclient init 2023-10-30T20:46:00.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.335+0000 7f0ceea2d640 10 monclient: init 2023-10-30T20:46:00.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 10 monclient: _reopen_session rank -1 2023-10-30T20:46:00.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:46:00.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ce80b12b0 0x7f0ce8140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:00.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 10 monclient(hunting): picked mon.noname-b con 0x7f0ce80b12b0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:46:00.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce80b4360 0x7f0ce8140ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:00.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 10 monclient(hunting): picked mon.noname-c con 0x7f0ce80b4360 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:46:00.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce81da230 0x7f0ce81da6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:00.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 10 monclient(hunting): picked mon.noname-a con 0x7f0ce81da230 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:46:00.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:00.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce80b0ee0 con 0x7f0ce81da230 2023-10-30T20:46:00.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:00.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce813f7a0 con 0x7f0ce80b12b0 2023-10-30T20:46:00.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:00.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce81dac20 con 0x7f0ce80b4360 2023-10-30T20:46:00.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 10 monclient(hunting): _renew_subs 2023-10-30T20:46:00.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.336+0000 7f0ceea2d640 10 monclient(hunting): authenticate will time out at 2883.625435s 2023-10-30T20:46:00.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.337+0000 7f0ce7fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ce80b12b0 0x7f0ce8140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:00.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.338+0000 7f0ce77fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce80b4360 0x7f0ce8140ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:00.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.338+0000 7f0ce77fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce80b4360 0x7f0ce8140ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:50314/0 (socket says 172.21.15.69:50314) 2023-10-30T20:46:00.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.338+0000 7f0ce77fe640 1 -- 172.21.15.69:0/3595226876 learned_addr learned my addr 172.21.15.69:0/3595226876 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:46:00.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.338+0000 7f0ce77fe640 10 monclient(hunting): get_auth_request con 0x7f0ce80b4360 auth_method 0 2023-10-30T20:46:00.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.338+0000 7f0ce77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:00.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.338+0000 7f0ce77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:00.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.338+0000 7f0ce77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:00.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.338+0000 7f0ce77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:00.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.338+0000 7f0ce77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:00.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.338+0000 7f0ce77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:00.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.340+0000 7f0ce7fff640 10 monclient(hunting): get_auth_request con 0x7f0ce80b12b0 auth_method 0 2023-10-30T20:46:00.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.340+0000 7f0ce7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:00.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.340+0000 7f0ce7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:00.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.340+0000 7f0ce7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:00.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.341+0000 7f0ce7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:00.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.341+0000 7f0ce7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:00.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.341+0000 7f0ce7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:00.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.342+0000 7f0ce77fe640 10 monclient(hunting): handle_auth_done global_id 18344 payload 824 2023-10-30T20:46:00.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.342+0000 7f0ce77fe640 1 -- 172.21.15.69:0/3595226876 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ce80b12b0 msgr2=0x7f0ce8140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:00.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.342+0000 7f0ce77fe640 1 --2- 172.21.15.69:0/3595226876 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ce80b12b0 0x7f0ce8140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.342+0000 7f0ce77fe640 1 -- 172.21.15.69:0/3595226876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce81da230 msgr2=0x7f0ce81da6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:00.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.342+0000 7f0ce77fe640 1 --2- 172.21.15.69:0/3595226876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce81da230 0x7f0ce81da6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.342+0000 7f0ce77fe640 10 monclient: _finish_hunting 0 2023-10-30T20:46:00.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.342+0000 7f0ce77fe640 1 monclient: found mon.noname-c 2023-10-30T20:46:00.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.342+0000 7f0ce77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:46:00.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.342+0000 7f0ce77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:46:00.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.342+0000 7f0ce77fe640 1 -- 172.21.15.69:0/3595226876 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce81d9d10 con 0x7f0ce80b4360 2023-10-30T20:46:00.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.344+0000 7f0ce77fe640 1 --2- 172.21.15.69:0/3595226876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce80b4360 0x7f0ce8140ba0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc00b440 tx=0x7f0cdc00b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:00.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.344+0000 7f0ce4ff9640 10 client.?.objecter ms_handle_connect 0x7f0ce80b4360 2023-10-30T20:46:00.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.344+0000 7f0ce4ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T20:46:00.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ce4ff9640 1 -- 172.21.15.69:0/3595226876 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc011890 con 0x7f0ce80b4360 2023-10-30T20:46:00.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ce4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:00.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ce4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:46:00.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ce4ff9640 10 monclient: dump: 2023-10-30T20:46:00.387 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:00.387 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:00.388 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.388 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.388 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:00.388 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:00.388 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:00.389 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:00.389 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:00.389 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:00.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ce4ff9640 10 monclient: _finish_auth 0 2023-10-30T20:46:00.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ce4ff9640 10 monclient: _check_auth_tickets 2023-10-30T20:46:00.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ce4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:46:00.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ceea2d640 5 monclient: authenticate success, global_id 18344 2023-10-30T20:46:00.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ceea2d640 10 monclient: _renew_subs 2023-10-30T20:46:00.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ceea2d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:46:00.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ceea2d640 1 -- 172.21.15.69:0/3595226876 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce8142c10 con 0x7f0ce80b4360 2023-10-30T20:46:00.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ceea2d640 10 client.18344.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:46:00.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ceea2d640 10 monclient: _renew_subs 2023-10-30T20:46:00.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ceea2d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:46:00.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ceea2d640 1 -- 172.21.15.69:0/3595226876 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce81410e0 con 0x7f0ce80b4360 2023-10-30T20:46:00.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.346+0000 7f0ceea2d640 1 librados: init done 2023-10-30T20:46:00.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.347+0000 7f0ceea2d640 10 client.18344.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:00.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.347+0000 7f0ceea2d640 15 client.18344.objecter _session_command_op_assign -1 1 2023-10-30T20:46:00.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.347+0000 7f0ceea2d640 20 client.18344.objecter _get_session osd=-1 returning homeless 2023-10-30T20:46:00.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.347+0000 7f0ceea2d640 10 client.18344.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:46:00.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.347+0000 7f0ceea2d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:46:00.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.347+0000 7f0ceea2d640 1 -- 172.21.15.69:0/3595226876 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0ce80a90e0 con 0x7f0ce80b4360 2023-10-30T20:46:00.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.348+0000 7f0ce4ff9640 1 -- 172.21.15.69:0/3595226876 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce80a90e0 con 0x7f0ce80b4360 2023-10-30T20:46:00.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.348+0000 7f0ce4ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:46:00.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.348+0000 7f0ce4ff9640 1 -- 172.21.15.69:0/3595226876 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc011c00 con 0x7f0ce80b4360 2023-10-30T20:46:00.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.348+0000 7f0ce4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:00.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.348+0000 7f0ce4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:46:00.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.348+0000 7f0ce4ff9640 10 monclient: dump: 2023-10-30T20:46:00.394 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:00.394 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:00.395 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.395 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.395 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:00.395 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:00.396 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.348+0000 7f0ce67fc640 4 set_mon_vals no callback set 2023-10-30T20:46:00.397 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:22.408 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:00.397 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:22.412 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:00.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.375+0000 7f0ce4ff9640 1 -- 172.21.15.69:0/3595226876 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f0cdc01a400 con 0x7f0ce80b4360 2023-10-30T20:46:00.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.375+0000 7f0ce4ff9640 1 --2- 172.21.15.69:0/3595226876 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0cd406c7f0 0x7f0cd406ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:00.398 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:22.435 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:00.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.375+0000 7f0ce7fff640 1 --2- 172.21.15.69:0/3595226876 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0cd406c7f0 0x7f0cd406ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:00.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.375+0000 7f0ce7fff640 10 monclient: get_auth_request con 0x7f0cd406c7f0 auth_method 0 2023-10-30T20:46:00.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.376+0000 7f0ce7fff640 1 --2- 172.21.15.69:0/3595226876 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0cd406c7f0 0x7f0cd406ecb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80017f0 tx=0x7f0cd8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:00.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.376+0000 7f0ce4ff9640 10 client.18344.objecter ms_handle_connect 0x7f0cd406c7f0 2023-10-30T20:46:00.401 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:22.428 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:00.404 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:22.432 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:00.419 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:22.446 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:00.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.423+0000 7f0ce4ff9640 1 -- 172.21.15.69:0/3595226876 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f0cdc058cb0 con 0x7f0ce80b4360 2023-10-30T20:46:00.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.423+0000 7f0ce4ff9640 10 client.18344.objecter ms_dispatch 0x7f0ce8000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:46:00.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.423+0000 7f0ce4ff9640 3 client.18344.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:46:00.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.423+0000 7f0ce4ff9640 3 client.18344.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:46:00.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0ce4ff9640 10 client.18344.objecter checking command 1 2023-10-30T20:46:00.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0ce4ff9640 1 --2- 172.21.15.69:0/3595226876 >> v2:172.21.15.183:6808/1361328034 conn(0x7f0cd4073c30 0x7f0cd40760d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:00.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0ce4ff9640 20 client.18344.objecter _get_session s=0x7f0cdc011c00 osd=7 3 2023-10-30T20:46:00.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0ce4ff9640 20 client.18344.objecter put_session s=0x7f0cdc011c00 osd=7 3 2023-10-30T20:46:00.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0ce4ff9640 15 client.18344.objecter _session_command_op_remove -1 1 2023-10-30T20:46:00.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0ce4ff9640 20 client.18344.objecter _get_session s=0x7f0cdc011c00 osd=7 3 2023-10-30T20:46:00.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0ce4ff9640 20 client.18344.objecter get_session s=0x7f0cdc011c00 osd=7 3 2023-10-30T20:46:00.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0ce4ff9640 15 client.18344.objecter _session_command_op_assign 7 1 2023-10-30T20:46:00.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0ce4ff9640 20 client.18344.objecter put_session s=0x7f0cdc011c00 osd=7 4 2023-10-30T20:46:00.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0ce4ff9640 10 client.18344.objecter _send_command 1 2023-10-30T20:46:00.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0ce4ff9640 1 -- 172.21.15.69:0/3595226876 --> v2:172.21.15.183:6808/1361328034 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0cdc011890 con 0x7f0cd4073c30 2023-10-30T20:46:00.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0ce4ff9640 20 client.18344.objecter dump_active .. 0 homeless 2023-10-30T20:46:00.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0ce4ff9640 1 -- 172.21.15.69:0/3595226876 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=904) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0cdc0548f0 con 0x7f0ce80b4360 2023-10-30T20:46:00.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0ce4ff9640 10 monclient: handle_get_version_reply finishing 1 version 904 2023-10-30T20:46:00.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.424+0000 7f0cecfca640 1 --2- 172.21.15.69:0/3595226876 >> v2:172.21.15.183:6808/1361328034 conn(0x7f0cd4073c30 0x7f0cd40760d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:00.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.425+0000 7f0cecfca640 10 monclient: get_auth_request con 0x7f0cd4073c30 auth_method 0 2023-10-30T20:46:00.430 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:22.487 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:00.430 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.427+0000 7f0cecfca640 1 --2- 172.21.15.69:0/3595226876 >> v2:172.21.15.183:6808/1361328034 conn(0x7f0cd4073c30 0x7f0cd40760d0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:00.430 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.427+0000 7f0ce4ff9640 10 client.18344.objecter ms_handle_connect 0x7f0cd4073c30 2023-10-30T20:46:00.433 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:22.492 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:00.447 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:22.506 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:00.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.450+0000 7f0ce4ff9640 1 -- 172.21.15.69:0/3595226876 <== osd.7 v2:172.21.15.183:6808/1361328034 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f0cdc011890 con 0x7f0cd4073c30 2023-10-30T20:46:00.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.450+0000 7f0ce4ff9640 10 client.18344.objecter ms_dispatch 0x7f0ce8000b90 command_reply(tid 1: 0 ) v1 2023-10-30T20:46:00.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.450+0000 7f0ce4ff9640 10 client.18344.objecter _finish_command 1 = system:0 2023-10-30T20:46:00.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.450+0000 7f0ce4ff9640 20 client.18344.objecter put_session s=0x7f0cdc011c00 osd=7 4 2023-10-30T20:46:00.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.450+0000 7f0ce4ff9640 15 client.18344.objecter _session_command_op_remove 7 1 2023-10-30T20:46:00.466 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.464+0000 7f0ceea2d640 10 client.18344.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T20:46:00.466 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.464+0000 7f0ceea2d640 15 client.18344.objecter _session_command_op_assign -1 2 2023-10-30T20:46:00.466 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.464+0000 7f0ceea2d640 20 client.18344.objecter _get_session s=0x7f0cdc011c00 osd=7 3 2023-10-30T20:46:00.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.464+0000 7f0ceea2d640 20 client.18344.objecter put_session s=0x7f0cdc011c00 osd=7 3 2023-10-30T20:46:00.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.464+0000 7f0ceea2d640 20 client.18344.objecter _get_session s=0x7f0cdc011c00 osd=7 3 2023-10-30T20:46:00.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.464+0000 7f0ceea2d640 15 client.18344.objecter _session_command_op_remove -1 2 2023-10-30T20:46:00.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.464+0000 7f0ceea2d640 20 client.18344.objecter get_session s=0x7f0cdc011c00 osd=7 3 2023-10-30T20:46:00.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.464+0000 7f0ceea2d640 15 client.18344.objecter _session_command_op_assign 7 2 2023-10-30T20:46:00.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.464+0000 7f0ceea2d640 20 client.18344.objecter put_session s=0x7f0cdc011c00 osd=7 4 2023-10-30T20:46:00.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.464+0000 7f0ceea2d640 10 client.18344.objecter _send_command 2 2023-10-30T20:46:00.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.464+0000 7f0ceea2d640 1 -- 172.21.15.69:0/3595226876 --> v2:172.21.15.183:6808/1361328034 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0ce80aa6f0 con 0x7f0cd4073c30 2023-10-30T20:46:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ce4ff9640 1 -- 172.21.15.69:0/3595226876 <== osd.7 v2:172.21.15.183:6808/1361328034 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f0ce002c7e0 con 0x7f0cd4073c30 2023-10-30T20:46:00.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ce4ff9640 10 client.18344.objecter ms_dispatch 0x7f0ce8000b90 command_reply(tid 2: 0 ) v1 2023-10-30T20:46:00.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ce4ff9640 10 client.18344.objecter _finish_command 2 = system:0 2023-10-30T20:46:00.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ce4ff9640 20 client.18344.objecter put_session s=0x7f0cdc011c00 osd=7 4 2023-10-30T20:46:00.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ce4ff9640 15 client.18344.objecter _session_command_op_remove 7 2 2023-10-30T20:46:00.482 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-10-30T20:46:00.482 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T20:46:00.482 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-10-30T20:46:00.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 10 librados: watch_flush enter 2023-10-30T20:46:00.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 10 librados: watch_flush exit 2023-10-30T20:46:00.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 10 client.18344.objecter close_session for osd.7 2023-10-30T20:46:00.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 1 -- 172.21.15.69:0/3595226876 >> v2:172.21.15.183:6808/1361328034 conn(0x7f0cd4073c30 msgr2=0x7f0cd40760d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:00.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 1 --2- 172.21.15.69:0/3595226876 >> v2:172.21.15.183:6808/1361328034 conn(0x7f0cd4073c30 0x7f0cd40760d0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 20 client.18344.objecter put_session s=0x7f0cdc011c00 osd=7 1 2023-10-30T20:46:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 20 client.18344.objecter shutdown clearing up homeless session... 2023-10-30T20:46:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 10 client.18344.objecter successfully canceled tick 2023-10-30T20:46:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 1 -- 172.21.15.69:0/3595226876 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0cd406c7f0 msgr2=0x7f0cd406ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 1 --2- 172.21.15.69:0/3595226876 >> v2:172.21.15.69:6800/1876307353 conn(0x7f0cd406c7f0 0x7f0cd406ecb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80017f0 tx=0x7f0cd8009040 comp rx=0 tx=0).stop 2023-10-30T20:46:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 10 monclient: shutdown 2023-10-30T20:46:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:46:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 1 -- 172.21.15.69:0/3595226876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce80b4360 msgr2=0x7f0ce8140ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 1 --2- 172.21.15.69:0/3595226876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce80b4360 0x7f0ce8140ba0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc00b440 tx=0x7f0cdc00b900 comp rx=0 tx=0).stop 2023-10-30T20:46:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0cecfca640 1 -- 172.21.15.69:0/3595226876 reap_dead start 2023-10-30T20:46:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 1 -- 172.21.15.69:0/3595226876 shutdown_connections 2023-10-30T20:46:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.479+0000 7f0ceea2d640 1 -- 172.21.15.69:0/3595226876 >> 172.21.15.69:0/3595226876 conn(0x7f0ce8065ed0 msgr2=0x7f0ce80662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:46:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.480+0000 7f0ceea2d640 1 -- 172.21.15.69:0/3595226876 shutdown_connections 2023-10-30T20:46:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.480+0000 7f0ceea2d640 1 -- 172.21.15.69:0/3595226876 wait complete. 2023-10-30T20:46:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.480+0000 7f0ceea2d640 1 librados: shutdown 2023-10-30T20:46:00.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-30T20:46:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.602+0000 7f8e4114e640 10 monclient: get_monmap_and_config 2023-10-30T20:46:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.602+0000 7f8e4114e640 10 monclient: build_initial_monmap 2023-10-30T20:46:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.602+0000 7f8e4114e640 10 monclient: monmap: 2023-10-30T20:46:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:46:00.605 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:00.605 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:46:00.603932+0000 2023-10-30T20:46:00.605 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:46:00.603932+0000 2023-10-30T20:46:00.605 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:46:00.605 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:00.606 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:46:00.606 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:46:00.606 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:46:00.606 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:46:00.606 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:46:00.607 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:00.608 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.603+0000 7f8e4114e640 1 Processor -- start 2023-10-30T20:46:00.608 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.603+0000 7f8e4114e640 1 -- start start 2023-10-30T20:46:00.608 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.603+0000 7f8e4114e640 10 monclient: init 2023-10-30T20:46:00.608 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.604+0000 7f8e4114e640 10 monclient: _reopen_session rank -1 2023-10-30T20:46:00.608 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.604+0000 7f8e4114e640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-10-30T20:46:00.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.604+0000 7f8e4114e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8e3c0b4350 0x7f8e3c0a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:00.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.604+0000 7f8e4114e640 10 monclient(hunting): picked mon.noname-c con 0x7f8e3c0b4350 addr v2:172.21.15.183:3300/0 2023-10-30T20:46:00.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.604+0000 7f8e4114e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e3c13eb50 0x7f8e3c13ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:00.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.604+0000 7f8e4114e640 10 monclient(hunting): picked mon.noname-b con 0x7f8e3c13eb50 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:46:00.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.604+0000 7f8e4114e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e3c144f50 addr v1:172.21.15.69:6789/0 2023-10-30T20:46:00.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.604+0000 7f8e4114e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3c0aa6e0 con 0x7f8e3c144f50 2023-10-30T20:46:00.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.604+0000 7f8e4114e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:00.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.604+0000 7f8e4114e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e3c0679f0 con 0x7f8e3c0b4350 2023-10-30T20:46:00.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.604+0000 7f8e4114e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:00.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.604+0000 7f8e4114e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c0b4730 con 0x7f8e3c13eb50 2023-10-30T20:46:00.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.604+0000 7f8e4114e640 10 monclient(hunting): _renew_subs 2023-10-30T20:46:00.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.604+0000 7f8e4114e640 10 monclient(hunting): authenticate will time out at 2883.893026s 2023-10-30T20:46:00.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.608+0000 7f8e3a59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e3c13eb50 0x7f8e3c13ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.615+0000 7f8e3a59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e3c13eb50 0x7f8e3c13ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:50330/0 (socket says 172.21.15.69:50330) 2023-10-30T20:46:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.615+0000 7f8e3a59c640 1 -- 172.21.15.69:0/4288986952 learned_addr learned my addr 172.21.15.69:0/4288986952 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:46:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.615+0000 7f8e3a59c640 10 monclient(hunting): get_auth_request con 0x7f8e3c13eb50 auth_method 0 2023-10-30T20:46:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.615+0000 7f8e3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.615+0000 7f8e3a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:00.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.615+0000 7f8e3a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:00.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.619+0000 7f8e3ad9d640 1 --2- 172.21.15.69:0/4288986952 >> v2:172.21.15.183:3300/0 conn(0x7f8e3c0b4350 0x7f8e3c0a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:00.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.620+0000 7f8e3ad9d640 10 monclient(hunting): get_auth_request con 0x7f8e3c0b4350 auth_method 0 2023-10-30T20:46:00.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.620+0000 7f8e3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:00.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.620+0000 7f8e3ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:00.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.620+0000 7f8e3ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:00.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.627+0000 7f8e3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:00.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.627+0000 7f8e3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:00.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.627+0000 7f8e3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:00.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.629+0000 7f8e3a59c640 10 monclient(hunting): handle_auth_done global_id 18347 payload 274 2023-10-30T20:46:00.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.629+0000 7f8e3a59c640 1 -- 172.21.15.69:0/4288986952 >> v2:172.21.15.183:3300/0 conn(0x7f8e3c0b4350 msgr2=0x7f8e3c0a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:46:00.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.629+0000 7f8e3a59c640 1 --2- 172.21.15.69:0/4288986952 >> v2:172.21.15.183:3300/0 conn(0x7f8e3c0b4350 0x7f8e3c0a83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.629+0000 7f8e3a59c640 1 -- 172.21.15.69:0/4288986952 >> v1:172.21.15.69:6789/0 conn(0x7f8e3c144f50 legacy=0x7f8e3c1453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:46:00.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.629+0000 7f8e3a59c640 10 monclient: _finish_hunting 0 2023-10-30T20:46:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.629+0000 7f8e3a59c640 1 monclient: found mon.noname-b 2023-10-30T20:46:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.629+0000 7f8e3a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:46:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.629+0000 7f8e3a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:46:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.629+0000 7f8e3a59c640 1 -- 172.21.15.69:0/4288986952 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3c0a8910 con 0x7f8e3c13eb50 2023-10-30T20:46:00.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.638+0000 7f8e3a59c640 1 --2- 172.21.15.69:0/4288986952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e3c13eb50 0x7f8e3c13ef70 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7f8e30009870 tx=0x7f8e3002b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a8cdaf860c52447 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:00.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e39d9b640 1 -- 172.21.15.69:0/4288986952 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e30031740 con 0x7f8e3c13eb50 2023-10-30T20:46:00.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e39d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:00.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e39d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:46:00.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e39d9b640 10 monclient: dump: 2023-10-30T20:46:00.682 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:00.682 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:00.682 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.683 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.683 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:00.683 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:00.683 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:00.683 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:00.684 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:00.684 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:00.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e39d9b640 10 monclient: _finish_auth 0 2023-10-30T20:46:00.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e39d9b640 10 monclient: _check_auth_tickets 2023-10-30T20:46:00.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e39d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:46:00.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e39d9b640 1 -- 172.21.15.69:0/4288986952 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e300318b0 con 0x7f8e3c13eb50 2023-10-30T20:46:00.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e39d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:46:00.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e39d9b640 1 -- 172.21.15.69:0/4288986952 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e30031c50 con 0x7f8e3c13eb50 2023-10-30T20:46:00.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e4114e640 5 monclient: authenticate success, global_id 18347 2023-10-30T20:46:00.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e4114e640 10 monclient: get_monmap_and_config success 2023-10-30T20:46:00.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e4114e640 4 set_mon_vals no callback set 2023-10-30T20:46:00.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e39d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:00.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e39d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:46:00.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e39d9b640 10 monclient: dump: 2023-10-30T20:46:00.686 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:00.686 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:00.687 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.687 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.687 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:00.687 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:00.687 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:00.687 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:00.688 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:00.688 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:00.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e4114e640 10 monclient: shutdown 2023-10-30T20:46:00.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e4114e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:46:00.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e4114e640 1 -- 172.21.15.69:0/4288986952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e3c13eb50 msgr2=0x7f8e3c13ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:00.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.680+0000 7f8e4114e640 1 --2- 172.21.15.69:0/4288986952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e3c13eb50 0x7f8e3c13ef70 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7f8e30009870 tx=0x7f8e3002b7e0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.681+0000 7f8e4114e640 1 -- 172.21.15.69:0/4288986952 shutdown_connections 2023-10-30T20:46:00.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.681+0000 7f8e4114e640 1 --2- 172.21.15.69:0/4288986952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e3c13eb50 0x7f8e3c13ef70 unknown :-1 s=CLOSED pgs=3213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.681+0000 7f8e4114e640 1 --2- 172.21.15.69:0/4288986952 >> v2:172.21.15.183:3300/0 conn(0x7f8e3c0b4350 0x7f8e3c0a83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.681+0000 7f8e4114e640 1 -- 172.21.15.69:0/4288986952 >> 172.21.15.69:0/4288986952 conn(0x7f8e3c065ea0 msgr2=0x7f8e3c0662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:46:00.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.681+0000 7f8e4114e640 1 -- 172.21.15.69:0/4288986952 shutdown_connections 2023-10-30T20:46:00.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.681+0000 7f8e4114e640 1 -- 172.21.15.69:0/4288986952 wait complete. 2023-10-30T20:46:00.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.681+0000 7f8e4114e640 10 monclient: build_initial_monmap 2023-10-30T20:46:00.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.681+0000 7f8e4114e640 10 monclient: monmap: 2023-10-30T20:46:00.692 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:46:00.693 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:46:00.693 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:46:00.693 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:46:00.693 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:46:00.693 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:46:00.694 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:46:00.694 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:46:00.694 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:46:00.694 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:00.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.682+0000 7f8e4114e640 1 librados: starting msgr at 2023-10-30T20:46:00.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.682+0000 7f8e4114e640 1 librados: starting objecter 2023-10-30T20:46:00.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.682+0000 7f8e4114e640 1 Processor -- start 2023-10-30T20:46:00.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.682+0000 7f8e4114e640 1 -- start start 2023-10-30T20:46:00.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.682+0000 7f8e4114e640 1 librados: setting wanted keys 2023-10-30T20:46:00.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.682+0000 7f8e4114e640 1 librados: calling monclient init 2023-10-30T20:46:00.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.682+0000 7f8e4114e640 10 monclient: init 2023-10-30T20:46:00.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 10 monclient: _reopen_session rank -1 2023-10-30T20:46:00.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T20:46:00.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e3c0b4350 0x7f8e3c0b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:00.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 10 monclient(hunting): picked mon.noname-b con 0x7f8e3c0b4350 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:46:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e3c13eb50 0x7f8e3c1dac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 10 monclient(hunting): picked mon.noname-c con 0x7f8e3c13eb50 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:46:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e3c144f50 0x7f8e3c13f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 10 monclient(hunting): picked mon.noname-a con 0x7f8e3c144f50 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:46:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:00.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c13f030 con 0x7f8e3c144f50 2023-10-30T20:46:00.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:00.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c1da510 con 0x7f8e3c0b4350 2023-10-30T20:46:00.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:00.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c1da7b0 con 0x7f8e3c13eb50 2023-10-30T20:46:00.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 10 monclient(hunting): _renew_subs 2023-10-30T20:46:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.683+0000 7f8e4114e640 10 monclient(hunting): authenticate will time out at 2883.971917s 2023-10-30T20:46:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.688+0000 7f8e3b59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e3c144f50 0x7f8e3c13f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.689+0000 7f8e3b59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e3c144f50 0x7f8e3c13f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50320/0 (socket says 172.21.15.69:50320) 2023-10-30T20:46:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.689+0000 7f8e3b59e640 1 -- 172.21.15.69:0/3117592508 learned_addr learned my addr 172.21.15.69:0/3117592508 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:46:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.689+0000 7f8e3b59e640 10 monclient(hunting): get_auth_request con 0x7f8e3c144f50 auth_method 0 2023-10-30T20:46:00.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.689+0000 7f8e3b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:00.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.689+0000 7f8e3b59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:00.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.689+0000 7f8e3b59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:00.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.691+0000 7f8e3ad9d640 1 --2- 172.21.15.69:0/3117592508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e3c0b4350 0x7f8e3c0b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:00.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.693+0000 7f8e3a59c640 1 --2- 172.21.15.69:0/3117592508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e3c13eb50 0x7f8e3c1dac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:00.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.694+0000 7f8e3b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:00.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.694+0000 7f8e3b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:00.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.694+0000 7f8e3b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:00.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.695+0000 7f8e3ad9d640 10 monclient(hunting): get_auth_request con 0x7f8e3c0b4350 auth_method 0 2023-10-30T20:46:00.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.695+0000 7f8e3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:00.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.695+0000 7f8e3ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:00.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.695+0000 7f8e3ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:00.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.696+0000 7f8e3a59c640 10 monclient(hunting): get_auth_request con 0x7f8e3c13eb50 auth_method 0 2023-10-30T20:46:00.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.696+0000 7f8e3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:00.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.696+0000 7f8e3a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:00.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.696+0000 7f8e3a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:00.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.699+0000 7f8e3b59e640 10 monclient(hunting): handle_auth_done global_id 16326 payload 824 2023-10-30T20:46:00.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.699+0000 7f8e3b59e640 1 -- 172.21.15.69:0/3117592508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e3c13eb50 msgr2=0x7f8e3c1dac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:00.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.699+0000 7f8e3b59e640 1 --2- 172.21.15.69:0/3117592508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e3c13eb50 0x7f8e3c1dac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.699+0000 7f8e3b59e640 1 -- 172.21.15.69:0/3117592508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e3c0b4350 msgr2=0x7f8e3c0b0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.699+0000 7f8e3b59e640 1 --2- 172.21.15.69:0/3117592508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e3c0b4350 0x7f8e3c0b0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.699+0000 7f8e3b59e640 10 monclient: _finish_hunting 0 2023-10-30T20:46:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.699+0000 7f8e3b59e640 1 monclient: found mon.noname-a 2023-10-30T20:46:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.699+0000 7f8e3b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:46:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.699+0000 7f8e3b59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-10-30T20:46:00.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.699+0000 7f8e3b59e640 1 -- 172.21.15.69:0/3117592508 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3c1db180 con 0x7f8e3c144f50 2023-10-30T20:46:00.706 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.699+0000 7f8e3ad9d640 1 --2- 172.21.15.69:0/3117592508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e3c0b4350 0x7f8e3c0b0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T20:46:00.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.707+0000 7f8e3b59e640 1 --2- 172.21.15.69:0/3117592508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e3c144f50 0x7f8e3c13f850 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400b9a0 tx=0x7f8e3400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:00.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.707+0000 7f8e237fe640 10 client.?.objecter ms_handle_connect 0x7f8e3c144f50 2023-10-30T20:46:00.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.707+0000 7f8e237fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:46:00.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e237fe640 1 -- 172.21.15.69:0/3117592508 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3400cb40 con 0x7f8e3c144f50 2023-10-30T20:46:00.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:00.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T20:46:00.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e237fe640 10 monclient: dump: 2023-10-30T20:46:00.721 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:00.721 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:00.722 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.722 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.722 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:00.722 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:00.722 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:00.722 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:00.723 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:00.723 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:00.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e237fe640 10 monclient: _finish_auth 0 2023-10-30T20:46:00.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e237fe640 10 monclient: _check_auth_tickets 2023-10-30T20:46:00.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:46:00.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e237fe640 1 -- 172.21.15.69:0/3117592508 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3400ccb0 con 0x7f8e3c144f50 2023-10-30T20:46:00.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e237fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:46:00.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e237fe640 1 -- 172.21.15.69:0/3117592508 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e34014680 con 0x7f8e3c144f50 2023-10-30T20:46:00.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e4114e640 5 monclient: authenticate success, global_id 16326 2023-10-30T20:46:00.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e4114e640 10 monclient: _renew_subs 2023-10-30T20:46:00.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e4114e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:46:00.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e4114e640 1 -- 172.21.15.69:0/3117592508 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e3c1db3e0 con 0x7f8e3c144f50 2023-10-30T20:46:00.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:00.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e3959a640 4 set_mon_vals no callback set 2023-10-30T20:46:00.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T20:46:00.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e237fe640 10 monclient: dump: 2023-10-30T20:46:00.725 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:00.726 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:00.726 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.726 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:00.726 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:00.726 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:00.727 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:00.727 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:00.727 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:00.727 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:00.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e4114e640 10 client.16326.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:46:00.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e4114e640 10 monclient: _renew_subs 2023-10-30T20:46:00.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e4114e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:46:00.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e4114e640 1 -- 172.21.15.69:0/3117592508 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e3c140080 con 0x7f8e3c144f50 2023-10-30T20:46:00.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.719+0000 7f8e4114e640 1 librados: init done 2023-10-30T20:46:00.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.720+0000 7f8e4114e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:00.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.720+0000 7f8e4114e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:00.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.720+0000 7f8e4114e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-10-30T20:46:00.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:00.720+0000 7f8e4114e640 1 -- 172.21.15.69:0/3117592508 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e3c1da960 con 0x7f8e3c144f50 2023-10-30T20:46:01.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.035+0000 7f8e237fe640 1 -- 172.21.15.69:0/3117592508 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f8e34003970 con 0x7f8e3c144f50 2023-10-30T20:46:01.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.036+0000 7f8e237fe640 1 --2- 172.21.15.69:0/3117592508 >> v2:172.21.15.69:6800/1876307353 conn(0x7f8e2806c810 0x7f8e2806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:01.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.036+0000 7f8e3ad9d640 1 --2- 172.21.15.69:0/3117592508 >> v2:172.21.15.69:6800/1876307353 conn(0x7f8e2806c810 0x7f8e2806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:01.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.036+0000 7f8e3ad9d640 10 monclient: get_auth_request con 0x7f8e2806c810 auth_method 0 2023-10-30T20:46:01.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.036+0000 7f8e3ad9d640 1 --2- 172.21.15.69:0/3117592508 >> v2:172.21.15.69:6800/1876307353 conn(0x7f8e2806c810 0x7f8e2806ecd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c006fd0 tx=0x7f8e2c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:01.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.036+0000 7f8e237fe640 10 client.16326.objecter ms_handle_connect 0x7f8e2806c810 2023-10-30T20:46:01.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.050+0000 7f8e237fe640 1 -- 172.21.15.69:0/3117592508 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f8e3400ce30 con 0x7f8e3c144f50 2023-10-30T20:46:01.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.050+0000 7f8e237fe640 10 client.16326.objecter ms_dispatch 0x7f8e3c000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:46:01.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.050+0000 7f8e237fe640 3 client.16326.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:46:01.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.050+0000 7f8e237fe640 3 client.16326.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:46:01.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.050+0000 7f8e237fe640 20 client.16326.objecter dump_active .. 0 homeless 2023-10-30T20:46:01.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.050+0000 7f8e237fe640 1 -- 172.21.15.69:0/3117592508 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8e3c140080 con 0x7f8e3c144f50 2023-10-30T20:46:01.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.050+0000 7f8e237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:01.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.050+0000 7f8e237fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:46:01.266 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.265+0000 7f8e4114e640 1 -- 172.21.15.69:0/3117592508 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e3c1db3e0 con 0x7f8e2806c810 2023-10-30T20:46:01.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.265+0000 7f8e237fe640 1 -- 172.21.15.69:0/3117592508 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8e3c1db3e0 con 0x7f8e2806c810 2023-10-30T20:46:01.267 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 7 to deep-scrub 2023-10-30T20:46:01.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 10 librados: watch_flush enter 2023-10-30T20:46:01.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 10 librados: watch_flush exit 2023-10-30T20:46:01.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 20 client.16326.objecter shutdown clearing up homeless session... 2023-10-30T20:46:01.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 10 client.16326.objecter successfully canceled tick 2023-10-30T20:46:01.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 1 -- 172.21.15.69:0/3117592508 >> v2:172.21.15.69:6800/1876307353 conn(0x7f8e2806c810 msgr2=0x7f8e2806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:01.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 1 --2- 172.21.15.69:0/3117592508 >> v2:172.21.15.69:6800/1876307353 conn(0x7f8e2806c810 0x7f8e2806ecd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c006fd0 tx=0x7f8e2c008040 comp rx=0 tx=0).stop 2023-10-30T20:46:01.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 10 monclient: shutdown 2023-10-30T20:46:01.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:46:01.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 1 -- 172.21.15.69:0/3117592508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e3c144f50 msgr2=0x7f8e3c13f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:01.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 1 --2- 172.21.15.69:0/3117592508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e3c144f50 0x7f8e3c13f850 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400b9a0 tx=0x7f8e3400be60 comp rx=0 tx=0).stop 2023-10-30T20:46:01.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 1 -- 172.21.15.69:0/3117592508 shutdown_connections 2023-10-30T20:46:01.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 1 --2- 172.21.15.69:0/3117592508 >> v2:172.21.15.69:6800/1876307353 conn(0x7f8e2806c810 0x7f8e2806ecd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:01.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 1 --2- 172.21.15.69:0/3117592508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e3c144f50 0x7f8e3c13f850 unknown :-1 s=CLOSED pgs=3187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:01.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 1 --2- 172.21.15.69:0/3117592508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e3c13eb50 0x7f8e3c1dac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:01.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 1 --2- 172.21.15.69:0/3117592508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e3c0b4350 0x7f8e3c0b0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:01.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 1 -- 172.21.15.69:0/3117592508 >> 172.21.15.69:0/3117592508 conn(0x7f8e3c065ea0 msgr2=0x7f8e3c13c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:46:01.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 1 -- 172.21.15.69:0/3117592508 shutdown_connections 2023-10-30T20:46:01.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 1 -- 172.21.15.69:0/3117592508 wait complete. 2023-10-30T20:46:01.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.267+0000 7f8e4114e640 1 librados: shutdown 2023-10-30T20:46:01.280 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T20:46:01.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.386+0000 7f53c15c9640 10 monclient: get_monmap_and_config 2023-10-30T20:46:01.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.386+0000 7f53c15c9640 10 monclient: build_initial_monmap 2023-10-30T20:46:01.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.386+0000 7f53c15c9640 10 monclient: monmap: 2023-10-30T20:46:01.388 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:46:01.388 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:01.389 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:46:01.387781+0000 2023-10-30T20:46:01.389 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:46:01.387781+0000 2023-10-30T20:46:01.389 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:46:01.389 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:01.389 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:46:01.389 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:46:01.390 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:46:01.390 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:46:01.390 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:46:01.390 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:01.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.387+0000 7f53c15c9640 1 Processor -- start 2023-10-30T20:46:01.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.387+0000 7f53c15c9640 1 -- start start 2023-10-30T20:46:01.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.387+0000 7f53c15c9640 10 monclient: init 2023-10-30T20:46:01.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.387+0000 7f53c15c9640 10 monclient: _reopen_session rank -1 2023-10-30T20:46:01.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53c15c9640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2023-10-30T20:46:01.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53c15c9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f53bc0b4390 0x7f53bc0a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:01.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53c15c9640 10 monclient(hunting): picked mon.noname-a con 0x7f53bc0b4390 addr v2:172.21.15.69:3300/0 2023-10-30T20:46:01.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53c15c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53bc1418a0 addr v1:172.21.15.183:6789/0 2023-10-30T20:46:01.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53c15c9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f53bc141ce0 0x7f53bc0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:01.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53c15c9640 10 monclient(hunting): picked mon.noname-c con 0x7f53bc141ce0 addr v2:172.21.15.183:3300/0 2023-10-30T20:46:01.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53c15c9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53bc0a8f60 con 0x7f53bc1418a0 2023-10-30T20:46:01.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53c15c9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:01.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53c15c9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53bc067a10 con 0x7f53bc0b4390 2023-10-30T20:46:01.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53c15c9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:01.395 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53c15c9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53bc0b4770 con 0x7f53bc141ce0 2023-10-30T20:46:01.395 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53c15c9640 10 monclient(hunting): _renew_subs 2023-10-30T20:46:01.395 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53c15c9640 10 monclient(hunting): authenticate will time out at 2884.676936s 2023-10-30T20:46:01.395 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53bb7fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f53bc141ce0 0x7f53bc0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:01.395 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53bb7fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f53bc141ce0 0x7f53bc0b2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:54170/0 (socket says 172.21.15.69:54170) 2023-10-30T20:46:01.395 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53bb7fe640 1 -- 172.21.15.69:0/2874475055 learned_addr learned my addr 172.21.15.69:0/2874475055 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:46:01.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.388+0000 7f53bb7fe640 10 monclient(hunting): get_auth_request con 0x7f53bc141ce0 auth_method 0 2023-10-30T20:46:01.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.389+0000 7f53bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:01.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.389+0000 7f53bb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:01.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.389+0000 7f53bb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:01.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.389+0000 7f53bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:01.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.389+0000 7f53bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:01.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.389+0000 7f53bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:01.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.390+0000 7f53baffd640 1 --2- 172.21.15.69:0/2874475055 >> v2:172.21.15.69:3300/0 conn(0x7f53bc0b4390 0x7f53bc0a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:01.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.391+0000 7f53bb7fe640 10 monclient(hunting): handle_auth_done global_id 21313 payload 274 2023-10-30T20:46:01.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.391+0000 7f53bb7fe640 1 -- 172.21.15.69:0/2874475055 >> v2:172.21.15.69:3300/0 conn(0x7f53bc0b4390 msgr2=0x7f53bc0a8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:46:01.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.391+0000 7f53bb7fe640 1 --2- 172.21.15.69:0/2874475055 >> v2:172.21.15.69:3300/0 conn(0x7f53bc0b4390 0x7f53bc0a8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:01.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.391+0000 7f53bb7fe640 1 -- 172.21.15.69:0/2874475055 >> v1:172.21.15.183:6789/0 conn(0x7f53bc1418a0 legacy=0x7f53bc0b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:46:01.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.391+0000 7f53bb7fe640 10 monclient: _finish_hunting 0 2023-10-30T20:46:01.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.391+0000 7f53bb7fe640 1 monclient: found mon.noname-c 2023-10-30T20:46:01.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.391+0000 7f53bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:46:01.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.391+0000 7f53bb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.183:3300/0 2023-10-30T20:46:01.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.391+0000 7f53bb7fe640 1 -- 172.21.15.69:0/2874475055 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc0aa3d0 con 0x7f53bc141ce0 2023-10-30T20:46:01.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.394+0000 7f53bb7fe640 1 --2- 172.21.15.69:0/2874475055 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc141ce0 0x7f53bc0b2210 secure :-1 s=READY pgs=5069 cs=0 l=1 rev1=1 crypto rx=0x7f53bc1428f0 tx=0x7f53ac02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f49c8a4cd0092e4e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:01.401 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53b9ffb640 1 -- 172.21.15.69:0/2874475055 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac02ca80 con 0x7f53bc141ce0 2023-10-30T20:46:01.401 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:01.401 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53b9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:46:01.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53b9ffb640 10 monclient: dump: 2023-10-30T20:46:01.402 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:01.402 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:01.402 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:01.402 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:01.403 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:01.403 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:01.403 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:01.403 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:01.403 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:01.404 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:01.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53b9ffb640 10 monclient: _finish_auth 0 2023-10-30T20:46:01.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53b9ffb640 10 monclient: _check_auth_tickets 2023-10-30T20:46:01.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:46:01.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53b9ffb640 1 -- 172.21.15.69:0/2874475055 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac02cbf0 con 0x7f53bc141ce0 2023-10-30T20:46:01.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53c15c9640 5 monclient: authenticate success, global_id 21313 2023-10-30T20:46:01.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53c15c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T20:46:01.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53b9ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:46:01.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53c15c9640 10 monclient: get_monmap_and_config success 2023-10-30T20:46:01.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53b9ffb640 1 -- 172.21.15.69:0/2874475055 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac0343f0 con 0x7f53bc141ce0 2023-10-30T20:46:01.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53c15c9640 4 set_mon_vals no callback set 2023-10-30T20:46:01.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:01.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53b9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:46:01.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.398+0000 7f53b9ffb640 10 monclient: dump: 2023-10-30T20:46:01.406 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:01.406 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:01.407 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:01.407 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:01.407 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:01.407 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:01.407 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:01.408 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:01.408 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:01.408 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:01.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.399+0000 7f53c15c9640 10 monclient: shutdown 2023-10-30T20:46:01.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.399+0000 7f53c15c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:46:01.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.399+0000 7f53c15c9640 1 -- 172.21.15.69:0/2874475055 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc141ce0 msgr2=0x7f53bc0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:01.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.399+0000 7f53c15c9640 1 --2- 172.21.15.69:0/2874475055 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc141ce0 0x7f53bc0b2210 secure :-1 s=READY pgs=5069 cs=0 l=1 rev1=1 crypto rx=0x7f53bc1428f0 tx=0x7f53ac02bee0 comp rx=0 tx=0).stop 2023-10-30T20:46:01.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.399+0000 7f53c15c9640 1 -- 172.21.15.69:0/2874475055 shutdown_connections 2023-10-30T20:46:01.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.399+0000 7f53c15c9640 1 --2- 172.21.15.69:0/2874475055 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc141ce0 0x7f53bc0b2210 unknown :-1 s=CLOSED pgs=5069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:01.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.399+0000 7f53c15c9640 1 --2- 172.21.15.69:0/2874475055 >> v2:172.21.15.69:3300/0 conn(0x7f53bc0b4390 0x7f53bc0a8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:01.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.399+0000 7f53c15c9640 1 -- 172.21.15.69:0/2874475055 >> 172.21.15.69:0/2874475055 conn(0x7f53bc065ea0 msgr2=0x7f53bc0662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:46:01.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.399+0000 7f53c15c9640 1 -- 172.21.15.69:0/2874475055 shutdown_connections 2023-10-30T20:46:01.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.399+0000 7f53c15c9640 1 -- 172.21.15.69:0/2874475055 wait complete. 2023-10-30T20:46:01.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.399+0000 7f53c15c9640 10 monclient: build_initial_monmap 2023-10-30T20:46:01.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.399+0000 7f53c15c9640 10 monclient: monmap: 2023-10-30T20:46:01.410 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:46:01.410 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:46:01.411 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:46:01.411 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:46:01.411 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:46:01.411 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:46:01.411 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:46:01.411 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:46:01.412 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:46:01.412 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:01.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 1 librados: starting msgr at 2023-10-30T20:46:01.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 1 librados: starting objecter 2023-10-30T20:46:01.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 1 Processor -- start 2023-10-30T20:46:01.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 1 -- start start 2023-10-30T20:46:01.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 1 librados: setting wanted keys 2023-10-30T20:46:01.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 1 librados: calling monclient init 2023-10-30T20:46:01.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 10 monclient: init 2023-10-30T20:46:01.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 10 monclient: _reopen_session rank -1 2023-10-30T20:46:01.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T20:46:01.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53bc0b4390 0x7f53bc143e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:01.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 10 monclient(hunting): picked mon.noname-a con 0x7f53bc0b4390 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:46:01.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53bc1418a0 0x7f53bc135630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:01.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 10 monclient(hunting): picked mon.noname-c con 0x7f53bc1418a0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:46:01.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc141ce0 0x7f53bc135b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:01.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 10 monclient(hunting): picked mon.noname-b con 0x7f53bc141ce0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:46:01.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:01.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0b3010 con 0x7f53bc0b4390 2023-10-30T20:46:01.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:01.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc143830 con 0x7f53bc141ce0 2023-10-30T20:46:01.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:01.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0b6a70 con 0x7f53bc1418a0 2023-10-30T20:46:01.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 10 monclient(hunting): _renew_subs 2023-10-30T20:46:01.419 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.400+0000 7f53c15c9640 10 monclient(hunting): authenticate will time out at 2884.689811s 2023-10-30T20:46:01.419 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.401+0000 7f53ba7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53bc1418a0 0x7f53bc135630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:01.419 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.401+0000 7f53bb7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc141ce0 0x7f53bc135b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:01.419 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.401+0000 7f53bb7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc141ce0 0x7f53bc135b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.69:54182/0 (socket says 172.21.15.69:54182) 2023-10-30T20:46:01.419 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.401+0000 7f53bb7fe640 1 -- 172.21.15.69:0/2211947424 learned_addr learned my addr 172.21.15.69:0/2211947424 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:46:01.419 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.401+0000 7f53bb7fe640 10 monclient(hunting): get_auth_request con 0x7f53bc141ce0 auth_method 0 2023-10-30T20:46:01.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.401+0000 7f53bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:01.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.401+0000 7f53bb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:01.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.401+0000 7f53bb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:01.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.401+0000 7f53ba7fc640 10 monclient(hunting): get_auth_request con 0x7f53bc1418a0 auth_method 0 2023-10-30T20:46:01.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.401+0000 7f53ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:01.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.401+0000 7f53ba7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:01.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.401+0000 7f53ba7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:01.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.402+0000 7f53bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:01.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.402+0000 7f53bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:01.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.402+0000 7f53bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:01.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.402+0000 7f53ba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:01.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.402+0000 7f53ba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:01.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.402+0000 7f53ba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:01.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.404+0000 7f53baffd640 1 --2- 172.21.15.69:0/2211947424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53bc0b4390 0x7f53bc143e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:01.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.404+0000 7f53bb7fe640 10 monclient(hunting): handle_auth_done global_id 21316 payload 824 2023-10-30T20:46:01.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.404+0000 7f53bb7fe640 1 -- 172.21.15.69:0/2211947424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53bc1418a0 msgr2=0x7f53bc135630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:01.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.404+0000 7f53bb7fe640 1 --2- 172.21.15.69:0/2211947424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53bc1418a0 0x7f53bc135630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:01.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.404+0000 7f53bb7fe640 1 -- 172.21.15.69:0/2211947424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53bc0b4390 msgr2=0x7f53bc143e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:01.423 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.404+0000 7f53bb7fe640 1 --2- 172.21.15.69:0/2211947424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53bc0b4390 0x7f53bc143e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:01.423 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.404+0000 7f53bb7fe640 10 monclient: _finish_hunting 0 2023-10-30T20:46:01.423 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.404+0000 7f53bb7fe640 1 monclient: found mon.noname-b 2023-10-30T20:46:01.423 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.404+0000 7f53bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:46:01.423 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.405+0000 7f53bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.183:3300/0 2023-10-30T20:46:01.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.405+0000 7f53bb7fe640 1 -- 172.21.15.69:0/2211947424 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc1d4130 con 0x7f53bc141ce0 2023-10-30T20:46:01.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.407+0000 7f53bb7fe640 1 --2- 172.21.15.69:0/2211947424 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc141ce0 0x7f53bc135b70 secure :-1 s=READY pgs=5070 cs=0 l=1 rev1=1 crypto rx=0x7f53ac036040 tx=0x7f53ac004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:01.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.407+0000 7f53a37fe640 10 client.?.objecter ms_handle_connect 0x7f53bc141ce0 2023-10-30T20:46:01.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.407+0000 7f53a37fe640 10 client.?.objecter resend_mon_ops 2023-10-30T20:46:01.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53a37fe640 1 -- 172.21.15.69:0/2211947424 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac03e450 con 0x7f53bc141ce0 2023-10-30T20:46:01.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:01.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:46:01.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53a37fe640 10 monclient: dump: 2023-10-30T20:46:01.425 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:01.425 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:01.425 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:01.425 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:01.426 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:01.426 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:01.426 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:01.426 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:01.426 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:01.427 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:01.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53a37fe640 10 monclient: _finish_auth 0 2023-10-30T20:46:01.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53a37fe640 10 monclient: _check_auth_tickets 2023-10-30T20:46:01.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:46:01.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53a37fe640 1 -- 172.21.15.69:0/2211947424 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac03e5e0 con 0x7f53bc141ce0 2023-10-30T20:46:01.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53a37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:46:01.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53a37fe640 1 -- 172.21.15.69:0/2211947424 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac03e9a0 con 0x7f53bc141ce0 2023-10-30T20:46:01.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53c15c9640 5 monclient: authenticate success, global_id 21316 2023-10-30T20:46:01.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53c15c9640 10 monclient: _renew_subs 2023-10-30T20:46:01.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53c15c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:46:01.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53c15c9640 1 -- 172.21.15.69:0/2211947424 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc143180 con 0x7f53bc141ce0 2023-10-30T20:46:01.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:01.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T20:46:01.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53a37fe640 10 monclient: dump: 2023-10-30T20:46:01.429 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:01.429 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:01.429 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:01.430 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:01.430 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:01.430 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:01.430 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:01.430 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:01.431 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:01.431 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:01.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53b97fa640 4 set_mon_vals no callback set 2023-10-30T20:46:01.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53c15c9640 10 client.21316.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:46:01.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53c15c9640 10 monclient: _renew_subs 2023-10-30T20:46:01.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53c15c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:46:01.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53c15c9640 1 -- 172.21.15.69:0/2211947424 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc1d6940 con 0x7f53bc141ce0 2023-10-30T20:46:01.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.410+0000 7f53c15c9640 1 librados: init done 2023-10-30T20:46:01.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.411+0000 7f53c15c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:01.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.411+0000 7f53c15c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:01.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.411+0000 7f53c15c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.183:3300/0 2023-10-30T20:46:01.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.411+0000 7f53c15c9640 1 -- 172.21.15.69:0/2211947424 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53bc0a9900 con 0x7f53bc141ce0 2023-10-30T20:46:01.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.429+0000 7f53a37fe640 1 -- 172.21.15.69:0/2211947424 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f53ac02ca90 con 0x7f53bc141ce0 2023-10-30T20:46:01.434 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.429+0000 7f53a37fe640 1 --2- 172.21.15.69:0/2211947424 >> v2:172.21.15.69:6800/1876307353 conn(0x7f53a806c810 0x7f53a806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:01.434 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.430+0000 7f53baffd640 1 --2- 172.21.15.69:0/2211947424 >> v2:172.21.15.69:6800/1876307353 conn(0x7f53a806c810 0x7f53a806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:01.434 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.430+0000 7f53baffd640 10 monclient: get_auth_request con 0x7f53a806c810 auth_method 0 2023-10-30T20:46:01.434 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.430+0000 7f53a37fe640 10 client.21316.objecter ms_handle_connect 0x7f53a806c810 2023-10-30T20:46:01.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.430+0000 7f53baffd640 1 --2- 172.21.15.69:0/2211947424 >> v2:172.21.15.69:6800/1876307353 conn(0x7f53a806c810 0x7f53a806ecd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f53b4006fd0 tx=0x7f53b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:01.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.431+0000 7f53a37fe640 1 -- 172.21.15.69:0/2211947424 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f53ac07b340 con 0x7f53bc141ce0 2023-10-30T20:46:01.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.431+0000 7f53a37fe640 10 client.21316.objecter ms_dispatch 0x7f53bc000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:46:01.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.431+0000 7f53a37fe640 3 client.21316.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:46:01.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.431+0000 7f53a37fe640 3 client.21316.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:46:01.436 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.431+0000 7f53a37fe640 20 client.21316.objecter dump_active .. 0 homeless 2023-10-30T20:46:01.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.664+0000 7f53a37fe640 1 -- 172.21.15.69:0/2211947424 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f53ac07ac40 con 0x7f53bc141ce0 2023-10-30T20:46:01.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.664+0000 7f53a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:01.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.664+0000 7f53a37fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:46:01.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.843+0000 7f53c15c9640 1 -- 172.21.15.69:0/2211947424 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f53bc142110 con 0x7f53a806c810 2023-10-30T20:46:01.849 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.848+0000 7f53a37fe640 1 -- 172.21.15.69:0/2211947424 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+150558 (secure 0 0 0) 0x7f53b400c040 con 0x7f53a806c810 2023-10-30T20:46:01.851 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:46:01.851 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-10-30T20:46:01.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 10 librados: watch_flush enter 2023-10-30T20:46:01.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 10 librados: watch_flush exit 2023-10-30T20:46:01.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 20 client.21316.objecter shutdown clearing up homeless session... 2023-10-30T20:46:01.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 10 client.21316.objecter successfully canceled tick 2023-10-30T20:46:01.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 1 -- 172.21.15.69:0/2211947424 >> v2:172.21.15.69:6800/1876307353 conn(0x7f53a806c810 msgr2=0x7f53a806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:01.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 1 --2- 172.21.15.69:0/2211947424 >> v2:172.21.15.69:6800/1876307353 conn(0x7f53a806c810 0x7f53a806ecd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f53b4006fd0 tx=0x7f53b4008040 comp rx=0 tx=0).stop 2023-10-30T20:46:01.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 10 monclient: shutdown 2023-10-30T20:46:01.854 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:46:01.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 1 -- 172.21.15.69:0/2211947424 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc141ce0 msgr2=0x7f53bc135b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:01.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 1 --2- 172.21.15.69:0/2211947424 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc141ce0 0x7f53bc135b70 secure :-1 s=READY pgs=5070 cs=0 l=1 rev1=1 crypto rx=0x7f53ac036040 tx=0x7f53ac004a40 comp rx=0 tx=0).stop 2023-10-30T20:46:01.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 1 -- 172.21.15.69:0/2211947424 shutdown_connections 2023-10-30T20:46:01.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 1 --2- 172.21.15.69:0/2211947424 >> v2:172.21.15.69:6800/1876307353 conn(0x7f53a806c810 0x7f53a806ecd0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:01.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 1 --2- 172.21.15.69:0/2211947424 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc141ce0 0x7f53bc135b70 unknown :-1 s=CLOSED pgs=5070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:01.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 1 --2- 172.21.15.69:0/2211947424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53bc1418a0 0x7f53bc135630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:01.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 1 --2- 172.21.15.69:0/2211947424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53bc0b4390 0x7f53bc143e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:01.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 1 -- 172.21.15.69:0/2211947424 >> 172.21.15.69:0/2211947424 conn(0x7f53bc065ea0 msgr2=0x7f53bc0b1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:46:01.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.850+0000 7f53c15c9640 1 -- 172.21.15.69:0/2211947424 shutdown_connections 2023-10-30T20:46:01.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.851+0000 7f53c15c9640 1 -- 172.21.15.69:0/2211947424 wait complete. 2023-10-30T20:46:01.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:01.851+0000 7f53c15c9640 1 librados: shutdown 2023-10-30T20:46:01.863 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1572,"stamp":"2023-10-30T20:46:00.504861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":545624,"kb_used_data":51908,"kb_used_omap":12,"kb_used_meta":493619,"kb_avail":754429096,"statfs":{"total":773094113280,"available":772535394304,"internally_reserved":0,"allocated":53153792,"data_stored":25973851,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12745,"internal_metadata":505466423},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.797593"},"pg_stats":[{"pgid":"10.16","version":"0'0","reported_seq":847,"reported_epoch":903,"state":"active+clean","last_fresh":"2023-10-30T20:45:50.919842+0000","last_change":"2023-10-30T20:25:23.971023+0000","last_active":"2023-10-30T20:45:50.919842+0000","last_peered":"2023-10-30T20:45:50.919842+0000","last_clean":"2023-10-30T20:45:50.919842+0000","last_became_active":"2023-10-30T20:25:23.962078+0000","last_became_peered":"2023-10-30T20:25:23.962078+0000","last_unstale":"2023-10-30T20:45:50.919842+0000","last_undegraded":"2023-10-30T20:45:50.919842+0000","last_fullsized":"2023-10-30T20:45:50.919842+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":872,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.725841+0000","last_change":"2023-10-30T20:25:24.870906+0000","last_active":"2023-10-30T20:45:55.725841+0000","last_peered":"2023-10-30T20:45:55.725841+0000","last_clean":"2023-10-30T20:45:55.725841+0000","last_became_active":"2023-10-30T20:25:24.861824+0000","last_became_peered":"2023-10-30T20:25:24.861824+0000","last_unstale":"2023-10-30T20:45:55.725841+0000","last_undegraded":"2023-10-30T20:45:55.725841+0000","last_fullsized":"2023-10-30T20:45:55.725841+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":852,"reported_epoch":903,"state":"active+clean","last_fresh":"2023-10-30T20:45:50.907109+0000","last_change":"2023-10-30T20:25:36.960712+0000","last_active":"2023-10-30T20:45:50.907109+0000","last_peered":"2023-10-30T20:45:50.907109+0000","last_clean":"2023-10-30T20:45:50.907109+0000","last_became_active":"2023-10-30T20:25:36.951760+0000","last_became_peered":"2023-10-30T20:25:36.951760+0000","last_unstale":"2023-10-30T20:45:50.907109+0000","last_undegraded":"2023-10-30T20:45:50.907109+0000","last_fullsized":"2023-10-30T20:45:50.907109+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:17:29.613581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":870,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.720775+0000","last_change":"2023-10-30T20:25:23.364080+0000","last_active":"2023-10-30T20:45:55.720775+0000","last_peered":"2023-10-30T20:45:55.720775+0000","last_clean":"2023-10-30T20:45:55.720775+0000","last_became_active":"2023-10-30T20:25:23.356998+0000","last_became_peered":"2023-10-30T20:25:23.356998+0000","last_unstale":"2023-10-30T20:45:55.720775+0000","last_undegraded":"2023-10-30T20:45:55.720775+0000","last_fullsized":"2023-10-30T20:45:55.720775+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":881,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.425071+0000","last_change":"2023-10-30T20:25:25.485385+0000","last_active":"2023-10-30T20:45:55.425071+0000","last_peered":"2023-10-30T20:45:55.425071+0000","last_clean":"2023-10-30T20:45:55.425071+0000","last_became_active":"2023-10-30T20:25:25.477142+0000","last_became_peered":"2023-10-30T20:25:25.477142+0000","last_unstale":"2023-10-30T20:45:55.425071+0000","last_undegraded":"2023-10-30T20:45:55.425071+0000","last_fullsized":"2023-10-30T20:45:55.425071+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":852,"reported_epoch":903,"state":"active+clean","last_fresh":"2023-10-30T20:45:50.912950+0000","last_change":"2023-10-30T20:25:23.846753+0000","last_active":"2023-10-30T20:45:50.912950+0000","last_peered":"2023-10-30T20:45:50.912950+0000","last_clean":"2023-10-30T20:45:50.912950+0000","last_became_active":"2023-10-30T20:25:23.840599+0000","last_became_peered":"2023-10-30T20:25:23.840599+0000","last_unstale":"2023-10-30T20:45:50.912950+0000","last_undegraded":"2023-10-30T20:45:50.912950+0000","last_fullsized":"2023-10-30T20:45:50.912950+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:46.666701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":881,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.414025+0000","last_change":"2023-10-30T20:25:28.699156+0000","last_active":"2023-10-30T20:45:55.414025+0000","last_peered":"2023-10-30T20:45:55.414025+0000","last_clean":"2023-10-30T20:45:55.414025+0000","last_became_active":"2023-10-30T20:25:28.691162+0000","last_became_peered":"2023-10-30T20:25:28.691162+0000","last_unstale":"2023-10-30T20:45:55.414025+0000","last_undegraded":"2023-10-30T20:45:55.414025+0000","last_fullsized":"2023-10-30T20:45:55.414025+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":881,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.422438+0000","last_change":"2023-10-30T20:25:36.297471+0000","last_active":"2023-10-30T20:45:55.422438+0000","last_peered":"2023-10-30T20:45:55.422438+0000","last_clean":"2023-10-30T20:45:55.422438+0000","last_became_active":"2023-10-30T20:25:36.286902+0000","last_became_peered":"2023-10-30T20:25:36.286902+0000","last_unstale":"2023-10-30T20:45:55.422438+0000","last_undegraded":"2023-10-30T20:45:55.422438+0000","last_fullsized":"2023-10-30T20:45:55.422438+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":859,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.711365+0000","last_change":"2023-10-30T20:25:25.388067+0000","last_active":"2023-10-30T20:45:55.711365+0000","last_peered":"2023-10-30T20:45:55.711365+0000","last_clean":"2023-10-30T20:45:55.711365+0000","last_became_active":"2023-10-30T20:25:25.367753+0000","last_became_peered":"2023-10-30T20:25:25.367753+0000","last_unstale":"2023-10-30T20:45:55.711365+0000","last_undegraded":"2023-10-30T20:45:55.711365+0000","last_fullsized":"2023-10-30T20:45:55.711365+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":857,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.721388+0000","last_change":"2023-10-30T20:25:33.376337+0000","last_active":"2023-10-30T20:45:55.721388+0000","last_peered":"2023-10-30T20:45:55.721388+0000","last_clean":"2023-10-30T20:45:55.721388+0000","last_became_active":"2023-10-30T20:25:33.362871+0000","last_became_peered":"2023-10-30T20:25:33.362871+0000","last_unstale":"2023-10-30T20:45:55.721388+0000","last_undegraded":"2023-10-30T20:45:55.721388+0000","last_fullsized":"2023-10-30T20:45:55.721388+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":858,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.711725+0000","last_change":"2023-10-30T20:25:35.098879+0000","last_active":"2023-10-30T20:45:55.711725+0000","last_peered":"2023-10-30T20:45:55.711725+0000","last_clean":"2023-10-30T20:45:55.711725+0000","last_became_active":"2023-10-30T20:25:35.047116+0000","last_became_peered":"2023-10-30T20:25:35.047116+0000","last_unstale":"2023-10-30T20:45:55.711725+0000","last_undegraded":"2023-10-30T20:45:55.711725+0000","last_fullsized":"2023-10-30T20:45:55.711725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":872,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.711102+0000","last_change":"2023-10-30T20:25:45.287217+0000","last_active":"2023-10-30T20:45:55.711102+0000","last_peered":"2023-10-30T20:45:55.711102+0000","last_clean":"2023-10-30T20:45:55.711102+0000","last_became_active":"2023-10-30T20:25:45.271565+0000","last_became_peered":"2023-10-30T20:25:45.271565+0000","last_unstale":"2023-10-30T20:45:55.711102+0000","last_undegraded":"2023-10-30T20:45:55.711102+0000","last_fullsized":"2023-10-30T20:45:55.711102+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":873,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.722505+0000","last_change":"2023-10-30T20:25:28.837575+0000","last_active":"2023-10-30T20:45:55.722505+0000","last_peered":"2023-10-30T20:45:55.722505+0000","last_clean":"2023-10-30T20:45:55.722505+0000","last_became_active":"2023-10-30T20:25:28.829991+0000","last_became_peered":"2023-10-30T20:25:28.829991+0000","last_unstale":"2023-10-30T20:45:55.722505+0000","last_undegraded":"2023-10-30T20:45:55.722505+0000","last_fullsized":"2023-10-30T20:45:55.722505+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":886,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:59.550815+0000","last_change":"2023-10-30T20:45:59.551856+0000","last_active":"2023-10-30T20:45:59.550815+0000","last_peered":"2023-10-30T20:45:59.550815+0000","last_clean":"2023-10-30T20:45:59.550815+0000","last_became_active":"2023-10-30T20:24:04.692769+0000","last_became_peered":"2023-10-30T20:24:04.692769+0000","last_unstale":"2023-10-30T20:45:59.550815+0000","last_undegraded":"2023-10-30T20:45:59.550815+0000","last_fullsized":"2023-10-30T20:45:59.550815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:59.549436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:59.549436+0000","last_clean_scrub_stamp":"2023-10-30T20:45:59.549436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T06:18:24.134116+0000","scrub_duration":0.076865441000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":848,"reported_epoch":903,"state":"active+clean","last_fresh":"2023-10-30T20:45:50.922082+0000","last_change":"2023-10-30T20:25:27.300642+0000","last_active":"2023-10-30T20:45:50.922082+0000","last_peered":"2023-10-30T20:45:50.922082+0000","last_clean":"2023-10-30T20:45:50.922082+0000","last_became_active":"2023-10-30T20:25:27.291944+0000","last_became_peered":"2023-10-30T20:25:27.291944+0000","last_unstale":"2023-10-30T20:45:50.922082+0000","last_undegraded":"2023-10-30T20:45:50.922082+0000","last_fullsized":"2023-10-30T20:45:50.922082+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":884,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:59.837045+0000","last_change":"2023-10-30T20:45:59.838090+0000","last_active":"2023-10-30T20:45:59.837045+0000","last_peered":"2023-10-30T20:45:59.837045+0000","last_clean":"2023-10-30T20:45:59.837045+0000","last_became_active":"2023-10-30T20:24:04.712006+0000","last_became_peered":"2023-10-30T20:24:04.712006+0000","last_unstale":"2023-10-30T20:45:59.837045+0000","last_undegraded":"2023-10-30T20:45:59.837045+0000","last_fullsized":"2023-10-30T20:45:59.837045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:59.835644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:59.835644+0000","last_clean_scrub_stamp":"2023-10-30T20:45:59.835644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:21:20.761019+0000","scrub_duration":0.035908962000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":850,"reported_epoch":903,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.350900+0000","last_change":"2023-10-30T20:25:39.862114+0000","last_active":"2023-10-30T20:45:55.350900+0000","last_peered":"2023-10-30T20:45:55.350900+0000","last_clean":"2023-10-30T20:45:55.350900+0000","last_became_active":"2023-10-30T20:25:39.852846+0000","last_became_peered":"2023-10-30T20:25:39.852846+0000","last_unstale":"2023-10-30T20:45:55.350900+0000","last_undegraded":"2023-10-30T20:45:55.350900+0000","last_fullsized":"2023-10-30T20:45:55.350900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":878,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.712674+0000","last_change":"2023-10-30T20:24:04.315228+0000","last_active":"2023-10-30T20:45:55.712674+0000","last_peered":"2023-10-30T20:45:55.712674+0000","last_clean":"2023-10-30T20:45:55.712674+0000","last_became_active":"2023-10-30T20:24:04.310065+0000","last_became_peered":"2023-10-30T20:24:04.310065+0000","last_unstale":"2023-10-30T20:45:55.712674+0000","last_undegraded":"2023-10-30T20:45:55.712674+0000","last_fullsized":"2023-10-30T20:45:55.712674+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":859,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.713330+0000","last_change":"2023-10-30T20:25:42.330893+0000","last_active":"2023-10-30T20:45:55.713330+0000","last_peered":"2023-10-30T20:45:55.713330+0000","last_clean":"2023-10-30T20:45:55.713330+0000","last_became_active":"2023-10-30T20:25:42.322505+0000","last_became_peered":"2023-10-30T20:25:42.322505+0000","last_unstale":"2023-10-30T20:45:55.713330+0000","last_undegraded":"2023-10-30T20:45:55.713330+0000","last_fullsized":"2023-10-30T20:45:55.713330+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":871,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:00.012998+0000","last_change":"2023-10-30T20:46:00.014043+0000","last_active":"2023-10-30T20:46:00.012998+0000","last_peered":"2023-10-30T20:46:00.012998+0000","last_clean":"2023-10-30T20:46:00.012998+0000","last_became_active":"2023-10-30T20:24:04.525673+0000","last_became_peered":"2023-10-30T20:24:04.525673+0000","last_unstale":"2023-10-30T20:46:00.012998+0000","last_undegraded":"2023-10-30T20:46:00.012998+0000","last_fullsized":"2023-10-30T20:46:00.012998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:00.011531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:00.011531+0000","last_clean_scrub_stamp":"2023-10-30T20:46:00.011531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:55:31.626108+0000","scrub_duration":0.034773421999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":867,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:00.336217+0000","last_change":"2023-10-30T20:46:00.338214+0000","last_active":"2023-10-30T20:46:00.336217+0000","last_peered":"2023-10-30T20:46:00.336217+0000","last_clean":"2023-10-30T20:46:00.336217+0000","last_became_active":"2023-10-30T20:25:33.242618+0000","last_became_peered":"2023-10-30T20:25:33.242618+0000","last_unstale":"2023-10-30T20:46:00.336217+0000","last_undegraded":"2023-10-30T20:46:00.336217+0000","last_fullsized":"2023-10-30T20:46:00.336217+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:00.329774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:00.329774+0000","last_clean_scrub_stamp":"2023-10-30T20:46:00.329774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:30:04.958207+0000","scrub_duration":0.033263122999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":878,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.697819+0000","last_change":"2023-10-30T20:24:04.525394+0000","last_active":"2023-10-30T20:45:55.697819+0000","last_peered":"2023-10-30T20:45:55.697819+0000","last_clean":"2023-10-30T20:45:55.697819+0000","last_became_active":"2023-10-30T20:24:04.512341+0000","last_became_peered":"2023-10-30T20:24:04.512341+0000","last_unstale":"2023-10-30T20:45:55.697819+0000","last_undegraded":"2023-10-30T20:45:55.697819+0000","last_fullsized":"2023-10-30T20:45:55.697819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":850,"reported_epoch":903,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.349383+0000","last_change":"2023-10-30T20:25:25.171687+0000","last_active":"2023-10-30T20:45:55.349383+0000","last_peered":"2023-10-30T20:45:55.349383+0000","last_clean":"2023-10-30T20:45:55.349383+0000","last_became_active":"2023-10-30T20:25:25.151444+0000","last_became_peered":"2023-10-30T20:25:25.151444+0000","last_unstale":"2023-10-30T20:45:55.349383+0000","last_undegraded":"2023-10-30T20:45:55.349383+0000","last_fullsized":"2023-10-30T20:45:55.349383+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":874,"reported_epoch":903,"state":"active+clean","last_fresh":"2023-10-30T20:45:50.898440+0000","last_change":"2023-10-30T20:24:05.504482+0000","last_active":"2023-10-30T20:45:50.898440+0000","last_peered":"2023-10-30T20:45:50.898440+0000","last_clean":"2023-10-30T20:45:50.898440+0000","last_became_active":"2023-10-30T20:24:05.475394+0000","last_became_peered":"2023-10-30T20:24:05.475394+0000","last_unstale":"2023-10-30T20:45:50.898440+0000","last_undegraded":"2023-10-30T20:45:50.898440+0000","last_fullsized":"2023-10-30T20:45:50.898440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:21:02.335096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":858,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.689105+0000","last_change":"2023-10-30T20:25:23.490954+0000","last_active":"2023-10-30T20:45:55.689105+0000","last_peered":"2023-10-30T20:45:55.689105+0000","last_clean":"2023-10-30T20:45:55.689105+0000","last_became_active":"2023-10-30T20:25:23.473157+0000","last_became_peered":"2023-10-30T20:25:23.473157+0000","last_unstale":"2023-10-30T20:45:55.689105+0000","last_undegraded":"2023-10-30T20:45:55.689105+0000","last_fullsized":"2023-10-30T20:45:55.689105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":878,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.693448+0000","last_change":"2023-10-30T20:24:04.284202+0000","last_active":"2023-10-30T20:45:55.693448+0000","last_peered":"2023-10-30T20:45:55.693448+0000","last_clean":"2023-10-30T20:45:55.693448+0000","last_became_active":"2023-10-30T20:24:04.254051+0000","last_became_peered":"2023-10-30T20:24:04.254051+0000","last_unstale":"2023-10-30T20:45:55.693448+0000","last_undegraded":"2023-10-30T20:45:55.693448+0000","last_fullsized":"2023-10-30T20:45:55.693448+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:24:02.536651+0000","last_clean_scrub_stamp":"2023-10-30T20:24:02.536651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":882,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.427010+0000","last_change":"2023-10-30T20:25:36.268959+0000","last_active":"2023-10-30T20:45:55.427010+0000","last_peered":"2023-10-30T20:45:55.427010+0000","last_clean":"2023-10-30T20:45:55.427010+0000","last_became_active":"2023-10-30T20:25:36.260481+0000","last_became_peered":"2023-10-30T20:25:36.260481+0000","last_unstale":"2023-10-30T20:45:55.427010+0000","last_undegraded":"2023-10-30T20:45:55.427010+0000","last_fullsized":"2023-10-30T20:45:55.427010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":886,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:58.542694+0000","last_change":"2023-10-30T20:45:58.543769+0000","last_active":"2023-10-30T20:45:58.542694+0000","last_peered":"2023-10-30T20:45:58.542694+0000","last_clean":"2023-10-30T20:45:58.542694+0000","last_became_active":"2023-10-30T20:24:04.671790+0000","last_became_peered":"2023-10-30T20:24:04.671790+0000","last_unstale":"2023-10-30T20:45:58.542694+0000","last_undegraded":"2023-10-30T20:45:58.542694+0000","last_fullsized":"2023-10-30T20:45:58.542694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:58.541256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:58.541256+0000","last_clean_scrub_stamp":"2023-10-30T20:45:58.541256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:47:21.915073+0000","scrub_duration":0.034256602999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":993,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.416175+0000","last_change":"2023-10-30T20:23:57.127569+0000","last_active":"2023-10-30T20:45:55.416175+0000","last_peered":"2023-10-30T20:45:55.416175+0000","last_clean":"2023-10-30T20:45:55.416175+0000","last_became_active":"2023-10-30T20:23:57.098872+0000","last_became_peered":"2023-10-30T20:23:57.098872+0000","last_unstale":"2023-10-30T20:45:55.416175+0000","last_undegraded":"2023-10-30T20:45:55.416175+0000","last_fullsized":"2023-10-30T20:45:55.416175+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:23:54.534733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:23:54.534733+0000","last_clean_scrub_stamp":"2023-10-30T20:23:54.534733+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"45'7","reported_seq":863,"reported_epoch":903,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.347586+0000","last_change":"2023-10-30T20:25:25.378575+0000","last_active":"2023-10-30T20:45:55.347586+0000","last_peered":"2023-10-30T20:45:55.347586+0000","last_clean":"2023-10-30T20:45:55.347586+0000","last_became_active":"2023-10-30T20:25:25.362636+0000","last_became_peered":"2023-10-30T20:25:25.362636+0000","last_unstale":"2023-10-30T20:45:55.347586+0000","last_undegraded":"2023-10-30T20:45:55.347586+0000","last_fullsized":"2023-10-30T20:45:55.347586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":868,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:59.300374+0000","last_change":"2023-10-30T20:45:59.302082+0000","last_active":"2023-10-30T20:45:59.300374+0000","last_peered":"2023-10-30T20:45:59.300374+0000","last_clean":"2023-10-30T20:45:59.300374+0000","last_became_active":"2023-10-30T20:25:23.607599+0000","last_became_peered":"2023-10-30T20:25:23.607599+0000","last_unstale":"2023-10-30T20:45:59.300374+0000","last_undegraded":"2023-10-30T20:45:59.300374+0000","last_fullsized":"2023-10-30T20:45:59.300374+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:59.298300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:59.298300+0000","last_clean_scrub_stamp":"2023-10-30T20:45:59.298300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:10:33.842120+0000","scrub_duration":0.034934029999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":871,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.711744+0000","last_change":"2023-10-30T20:25:28.335295+0000","last_active":"2023-10-30T20:45:55.711744+0000","last_peered":"2023-10-30T20:45:55.711744+0000","last_clean":"2023-10-30T20:45:55.711744+0000","last_became_active":"2023-10-30T20:25:28.326296+0000","last_became_peered":"2023-10-30T20:25:28.326296+0000","last_unstale":"2023-10-30T20:45:55.711744+0000","last_undegraded":"2023-10-30T20:45:55.711744+0000","last_fullsized":"2023-10-30T20:45:55.711744+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":858,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.694335+0000","last_change":"2023-10-30T20:25:20.677355+0000","last_active":"2023-10-30T20:45:55.694335+0000","last_peered":"2023-10-30T20:45:55.694335+0000","last_clean":"2023-10-30T20:45:55.694335+0000","last_became_active":"2023-10-30T20:25:20.671965+0000","last_became_peered":"2023-10-30T20:25:20.671965+0000","last_unstale":"2023-10-30T20:45:55.694335+0000","last_undegraded":"2023-10-30T20:45:55.694335+0000","last_fullsized":"2023-10-30T20:45:55.694335+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":871,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.718262+0000","last_change":"2023-10-30T20:25:28.987986+0000","last_active":"2023-10-30T20:45:55.718262+0000","last_peered":"2023-10-30T20:45:55.718262+0000","last_clean":"2023-10-30T20:45:55.718262+0000","last_became_active":"2023-10-30T20:25:28.979668+0000","last_became_peered":"2023-10-30T20:25:28.979668+0000","last_unstale":"2023-10-30T20:45:55.718262+0000","last_undegraded":"2023-10-30T20:45:55.718262+0000","last_fullsized":"2023-10-30T20:45:55.718262+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":871,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.707427+0000","last_change":"2023-10-30T20:25:23.069394+0000","last_active":"2023-10-30T20:45:55.707427+0000","last_peered":"2023-10-30T20:45:55.707427+0000","last_clean":"2023-10-30T20:45:55.707427+0000","last_became_active":"2023-10-30T20:25:23.058176+0000","last_became_peered":"2023-10-30T20:25:23.058176+0000","last_unstale":"2023-10-30T20:45:55.707427+0000","last_undegraded":"2023-10-30T20:45:55.707427+0000","last_fullsized":"2023-10-30T20:45:55.707427+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":873,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.724171+0000","last_change":"2023-10-30T20:25:28.823327+0000","last_active":"2023-10-30T20:45:55.724171+0000","last_peered":"2023-10-30T20:45:55.724171+0000","last_clean":"2023-10-30T20:45:55.724171+0000","last_became_active":"2023-10-30T20:25:28.798013+0000","last_became_peered":"2023-10-30T20:25:28.798013+0000","last_unstale":"2023-10-30T20:45:55.724171+0000","last_undegraded":"2023-10-30T20:45:55.724171+0000","last_fullsized":"2023-10-30T20:45:55.724171+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":873,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.715725+0000","last_change":"2023-10-30T20:25:23.723655+0000","last_active":"2023-10-30T20:45:55.715725+0000","last_peered":"2023-10-30T20:45:55.715725+0000","last_clean":"2023-10-30T20:45:55.715725+0000","last_became_active":"2023-10-30T20:25:23.709112+0000","last_became_peered":"2023-10-30T20:25:23.709112+0000","last_unstale":"2023-10-30T20:45:55.715725+0000","last_undegraded":"2023-10-30T20:45:55.715725+0000","last_fullsized":"2023-10-30T20:45:55.715725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":852,"reported_epoch":903,"state":"active+clean","last_fresh":"2023-10-30T20:45:50.900916+0000","last_change":"2023-10-30T20:25:28.781780+0000","last_active":"2023-10-30T20:45:50.900916+0000","last_peered":"2023-10-30T20:45:50.900916+0000","last_clean":"2023-10-30T20:45:50.900916+0000","last_became_active":"2023-10-30T20:25:28.762953+0000","last_became_peered":"2023-10-30T20:25:28.762953+0000","last_unstale":"2023-10-30T20:45:50.900916+0000","last_undegraded":"2023-10-30T20:45:50.900916+0000","last_fullsized":"2023-10-30T20:45:50.900916+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:09.371593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":870,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.694666+0000","last_change":"2023-10-30T20:25:21.917941+0000","last_active":"2023-10-30T20:45:55.694666+0000","last_peered":"2023-10-30T20:45:55.694666+0000","last_clean":"2023-10-30T20:45:55.694666+0000","last_became_active":"2023-10-30T20:25:21.911569+0000","last_became_peered":"2023-10-30T20:25:21.911569+0000","last_unstale":"2023-10-30T20:45:55.694666+0000","last_undegraded":"2023-10-30T20:45:55.694666+0000","last_fullsized":"2023-10-30T20:45:55.694666+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":877,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.710201+0000","last_change":"2023-10-30T20:29:14.877146+0000","last_active":"2023-10-30T20:45:55.710201+0000","last_peered":"2023-10-30T20:45:55.710201+0000","last_clean":"2023-10-30T20:45:55.710201+0000","last_became_active":"2023-10-30T20:29:14.860294+0000","last_became_peered":"2023-10-30T20:29:14.860294+0000","last_unstale":"2023-10-30T20:45:55.710201+0000","last_undegraded":"2023-10-30T20:45:55.710201+0000","last_fullsized":"2023-10-30T20:45:55.710201+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":881,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.420035+0000","last_change":"2023-10-30T20:25:29.021159+0000","last_active":"2023-10-30T20:45:55.420035+0000","last_peered":"2023-10-30T20:45:55.420035+0000","last_clean":"2023-10-30T20:45:55.420035+0000","last_became_active":"2023-10-30T20:25:29.004167+0000","last_became_peered":"2023-10-30T20:25:29.004167+0000","last_unstale":"2023-10-30T20:45:55.420035+0000","last_undegraded":"2023-10-30T20:45:55.420035+0000","last_fullsized":"2023-10-30T20:45:55.420035+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477003,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59864,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311976,"statfs":{"total":96636764160,"available":96575463424,"internally_reserved":0,"allocated":6635520,"data_stored":3147947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":25.940000000000001,"5min":18.199999999999999,"15min":16.863},"min":{"1min":5.1040000000000001,"5min":5.0529999999999999,"15min":3.9740000000000002},"max":{"1min":156.071,"5min":222.095,"15min":244.999},"last":14.005000000000001},{"interface":"front","average":{"1min":32.825000000000003,"5min":20.239000000000001,"15min":18.428000000000001},"min":{"1min":5.1020000000000003,"5min":4.9279999999999999,"15min":4.2539999999999996},"max":{"1min":219.797,"5min":219.797,"15min":219.797},"last":10.189}]},{"osd":1,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":25.375,"5min":20.100999999999999,"15min":20.373000000000001},"min":{"1min":9.6869999999999994,"5min":5.6500000000000004,"15min":4.5940000000000003},"max":{"1min":143.87200000000001,"5min":231.85499999999999,"15min":256.815},"last":5.2309999999999999},{"interface":"front","average":{"1min":22.506,"5min":19.602,"15min":22.550999999999998},"min":{"1min":7.298,"5min":5.7320000000000002,"15min":5.726},"max":{"1min":122.70399999999999,"5min":237.88300000000001,"15min":424.57999999999998},"last":9.3729999999999993}]},{"osd":2,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":34.981999999999999,"5min":24.867999999999999,"15min":24.663},"min":{"1min":9.7789999999999999,"5min":6.1539999999999999,"15min":6.1539999999999999},"max":{"1min":267.81099999999998,"5min":267.81099999999998,"15min":835.96799999999996},"last":8.2739999999999991},{"interface":"front","average":{"1min":37.021999999999998,"5min":26.030999999999999,"15min":23.010000000000002},"min":{"1min":10.771000000000001,"5min":10.173,"15min":8.2479999999999993},"max":{"1min":244.50399999999999,"5min":247.178,"15min":261.50900000000001},"last":11.957000000000001}]},{"osd":4,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":29.966000000000001,"5min":27.501000000000001,"15min":23.704999999999998},"min":{"1min":6.8390000000000004,"5min":6.625,"15min":5.5190000000000001},"max":{"1min":196.904,"5min":247.904,"15min":247.904},"last":7.5430000000000001},{"interface":"front","average":{"1min":29.170999999999999,"5min":25.465,"15min":24.071000000000002},"min":{"1min":7.9900000000000002,"5min":6.8620000000000001,"15min":6.3840000000000003},"max":{"1min":210.88200000000001,"5min":235.64099999999999,"15min":235.64099999999999},"last":14.888999999999999}]},{"osd":5,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":35.137999999999998,"5min":26.385999999999999,"15min":25.082999999999998},"min":{"1min":8.4100000000000001,"5min":6.726,"15min":6.726},"max":{"1min":198.76300000000001,"5min":233.28999999999999,"15min":233.28999999999999},"last":11.371},{"interface":"front","average":{"1min":46.277000000000001,"5min":31.02,"15min":24.657},"min":{"1min":11.351000000000001,"5min":10.645,"15min":7.883},"max":{"1min":227.28299999999999,"5min":244.23500000000001,"15min":244.23500000000001},"last":17.327000000000002}]},{"osd":6,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":41.406999999999996,"5min":30.888000000000002,"15min":27.672999999999998},"min":{"1min":11.547000000000001,"5min":9.1989999999999998,"15min":9.1989999999999998},"max":{"1min":269.15600000000001,"5min":269.15600000000001,"15min":269.15600000000001},"last":18.945},{"interface":"front","average":{"1min":32.853999999999999,"5min":24.702999999999999,"15min":22.295999999999999},"min":{"1min":6.093,"5min":6.093,"15min":6.093},"max":{"1min":262.53500000000003,"5min":262.53500000000003,"15min":262.53500000000003},"last":15.75}]},{"osd":7,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":29.765999999999998,"5min":27.036000000000001,"15min":24.699000000000002},"min":{"1min":9.7669999999999995,"5min":8.5570000000000004,"15min":7.3799999999999999},"max":{"1min":222.32400000000001,"5min":264.48000000000002,"15min":264.48000000000002},"last":17.809000000000001},{"interface":"front","average":{"1min":31.981999999999999,"5min":27.591000000000001,"15min":26.734999999999999},"min":{"1min":9.2919999999999998,"5min":8.9700000000000006,"15min":8.9700000000000006},"max":{"1min":233.99700000000001,"5min":245.042,"15min":245.042},"last":12.784000000000001}]}]},{"osd":1,"up_from":16,"seq":68719477003,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64064,"kb_used_data":6516,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94307776,"statfs":{"total":96636764160,"available":96571162624,"internally_reserved":0,"allocated":6672384,"data_stored":3545787,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.762,"5min":15.228999999999999,"15min":16.553999999999998},"min":{"1min":4.1130000000000004,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":98.356999999999999,"5min":244.65000000000001,"15min":244.65000000000001},"last":5.1449999999999996},{"interface":"front","average":{"1min":11.536,"5min":12.617000000000001,"15min":18.32},"min":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":3.847},"max":{"1min":39.68,"5min":65.853999999999999,"15min":407.38900000000001},"last":8.9540000000000006}]},{"osd":2,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":18.402000000000001,"5min":16.823,"15min":19.931000000000001},"min":{"1min":7.5389999999999997,"5min":7.367,"15min":4.1500000000000004},"max":{"1min":80.688000000000002,"5min":113.649,"15min":297.26499999999999},"last":9.6699999999999999},{"interface":"front","average":{"1min":23.346,"5min":19.442,"15min":20.483000000000001},"min":{"1min":9.8040000000000003,"5min":8.5690000000000008,"15min":6.5469999999999997},"max":{"1min":148.03999999999999,"5min":148.03999999999999,"15min":167.12100000000001},"last":13.693}]},{"osd":3,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":17.946000000000002,"5min":19.791,"15min":21.096},"min":{"1min":8.8569999999999993,"5min":7.8029999999999999,"15min":6.5990000000000002},"max":{"1min":84.090999999999994,"5min":150.166,"15min":213.971},"last":18.065000000000001},{"interface":"front","average":{"1min":19.449000000000002,"5min":22.609000000000002,"15min":24.210999999999999},"min":{"1min":12.093999999999999,"5min":9.8409999999999993,"15min":8.1449999999999996},"max":{"1min":74.275999999999996,"5min":146.15899999999999,"15min":223.49199999999999},"last":20.128}]},{"osd":4,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":13.090999999999999,"5min":20.369,"15min":21.076000000000001},"min":{"1min":8.1349999999999998,"5min":6.8659999999999997,"15min":5.9809999999999999},"max":{"1min":30.533000000000001,"5min":200.98400000000001,"15min":215.435},"last":10.35},{"interface":"front","average":{"1min":17.289000000000001,"5min":23.798999999999999,"15min":25.364000000000001},"min":{"1min":12.874000000000001,"5min":11.516999999999999,"15min":8.9939999999999998},"max":{"1min":26.600000000000001,"5min":192.69200000000001,"15min":197.34},"last":19.170999999999999}]},{"osd":5,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":13.349,"5min":19.747,"15min":19.356000000000002},"min":{"1min":7.4020000000000001,"5min":6.4580000000000002,"15min":5.1390000000000002},"max":{"1min":62.798000000000002,"5min":209.35499999999999,"15min":241.798},"last":11.068},{"interface":"front","average":{"1min":17.783000000000001,"5min":20.292000000000002,"15min":21.498000000000001},"min":{"1min":9.5739999999999998,"5min":6.6230000000000002,"15min":6.6230000000000002},"max":{"1min":68.882000000000005,"5min":118.65600000000001,"15min":248.124},"last":12.824}]},{"osd":6,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.948,"5min":20.902000000000001,"15min":23.236999999999998},"min":{"1min":10.205,"5min":7.7220000000000004,"15min":6.9880000000000004},"max":{"1min":37.037999999999997,"5min":204.71899999999999,"15min":406.72399999999999},"last":14.444000000000001},{"interface":"front","average":{"1min":28.690999999999999,"5min":25.748000000000001,"15min":25.416},"min":{"1min":11.704000000000001,"5min":10.397,"15min":8.4749999999999996},"max":{"1min":179.98099999999999,"5min":179.98099999999999,"15min":179.98099999999999},"last":21.099}]},{"osd":7,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":14.643000000000001,"5min":18.478999999999999,"15min":23.765000000000001},"min":{"1min":6.0410000000000004,"5min":6.0410000000000004,"15min":6.0110000000000001},"max":{"1min":58.841999999999999,"5min":188.792,"15min":249.88300000000001},"last":15.141},{"interface":"front","average":{"1min":19.361000000000001,"5min":21.233000000000001,"15min":25.628},"min":{"1min":12.742000000000001,"5min":8.8559999999999999,"15min":7.1360000000000001},"max":{"1min":59.718000000000004,"5min":134.89400000000001,"15min":275.36700000000002},"last":16.98}]}]},{"osd":0,"up_from":16,"seq":68719477003,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59860,"kb_used_data":6472,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311980,"statfs":{"total":96636764160,"available":96575467520,"internally_reserved":0,"allocated":6627328,"data_stored":3142537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":14.162000000000001,"5min":13.208,"15min":18.428000000000001},"min":{"1min":5.508,"5min":4.1740000000000004,"15min":4.0819999999999999},"max":{"1min":64.745999999999995,"5min":145.857,"15min":341.85300000000001},"last":14.141999999999999},{"interface":"front","average":{"1min":15.507,"5min":16.523,"15min":18.965},"min":{"1min":5.9400000000000004,"5min":5.798,"15min":5.1669999999999998},"max":{"1min":71.578000000000003,"5min":196.148,"15min":243.62100000000001},"last":10.768000000000001}]},{"osd":2,"last update":"Mon Oct 30 20:44:59 2023","interfaces":[{"interface":"back","average":{"1min":18.457000000000001,"5min":19.640999999999998,"15min":31.206},"min":{"1min":6.5940000000000003,"5min":4.6449999999999996,"15min":4.6449999999999996},"max":{"1min":68.856999999999999,"5min":199.71100000000001,"15min":580.43100000000004},"last":7.431},{"interface":"front","average":{"1min":15.906000000000001,"5min":18.309000000000001,"15min":31.786999999999999},"min":{"1min":6.6920000000000002,"5min":6.6920000000000002,"15min":6.1420000000000003},"max":{"1min":55.250999999999998,"5min":197.125,"15min":1090.171},"last":11.465999999999999}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.609,"5min":16.076000000000001,"15min":16.977},"min":{"1min":6.2199999999999998,"5min":5.298,"15min":5.298},"max":{"1min":76.861000000000004,"5min":148.31299999999999,"15min":167.607},"last":6.7949999999999999},{"interface":"front","average":{"1min":30.317,"5min":25.641999999999999,"15min":24.917999999999999},"min":{"1min":7.1719999999999997,"5min":7.1719999999999997,"15min":6.4370000000000003},"max":{"1min":290.19600000000003,"5min":290.19600000000003,"15min":318.245},"last":13.313000000000001}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":24.504000000000001,"5min":20.890000000000001,"15min":20.295000000000002},"min":{"1min":7.234,"5min":7.1059999999999999,"15min":5.8170000000000002},"max":{"1min":116.98399999999999,"5min":174.67699999999999,"15min":178.642},"last":9.0619999999999994},{"interface":"front","average":{"1min":30.776,"5min":28.367999999999999,"15min":27.843},"min":{"1min":8.3770000000000007,"5min":8.1010000000000009,"15min":6.2960000000000003},"max":{"1min":153.124,"5min":219.14699999999999,"15min":242.06800000000001},"last":16.202000000000002}]},{"osd":5,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.25,"5min":23.315999999999999,"15min":22.43},"min":{"1min":9.3379999999999992,"5min":8.3480000000000008,"15min":6.8099999999999996},"max":{"1min":68.546999999999997,"5min":170.024,"15min":170.024},"last":15.691000000000001},{"interface":"front","average":{"1min":25.843,"5min":23.294,"15min":22.381},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.8280000000000003},"max":{"1min":138.041,"5min":177.41800000000001,"15min":177.41800000000001},"last":10.007999999999999}]},{"osd":6,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":27.763999999999999,"5min":29.074999999999999,"15min":30.167000000000002},"min":{"1min":10.817,"5min":8.7599999999999998,"15min":8.2590000000000003},"max":{"1min":151.73500000000001,"5min":189.21899999999999,"15min":330.37799999999999},"last":18.693000000000001},{"interface":"front","average":{"1min":19.260000000000002,"5min":20.901,"15min":24.004000000000001},"min":{"1min":6.8410000000000002,"5min":6.8410000000000002,"15min":6.8410000000000002},"max":{"1min":73.170000000000002,"5min":169.447,"15min":201.50299999999999},"last":16.920000000000002}]},{"osd":7,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":20.75,"5min":23.949999999999999,"15min":27.219999999999999},"min":{"1min":11.298,"5min":10.81,"15min":9.6820000000000004},"max":{"1min":70.683000000000007,"5min":175.667,"15min":268.18900000000002},"last":17.835999999999999},{"interface":"front","average":{"1min":21.437999999999999,"5min":21.204999999999998,"15min":25.626000000000001},"min":{"1min":8.8689999999999998,"5min":8.8689999999999998,"15min":7.3559999999999999},"max":{"1min":77.927000000000007,"5min":173.441,"15min":228.495},"last":12.726000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509707,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59868,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311972,"statfs":{"total":96636764160,"available":96575459328,"internally_reserved":0,"allocated":6635520,"data_stored":3147947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:49 2023","interfaces":[{"interface":"back","average":{"1min":15.789,"5min":12.747,"15min":13.856999999999999},"min":{"1min":4.3819999999999997,"5min":4.0179999999999998,"15min":3.7080000000000002},"max":{"1min":66.393000000000001,"5min":73.316000000000003,"15min":151.00299999999999},"last":9.2569999999999997},{"interface":"front","average":{"1min":10.926,"5min":11.757,"15min":15.211},"min":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"max":{"1min":53.823999999999998,"5min":75.060000000000002,"15min":146.49299999999999},"last":7.7030000000000003}]},{"osd":1,"last update":"Mon Oct 30 20:45:49 2023","interfaces":[{"interface":"back","average":{"1min":14.625,"5min":16.163,"15min":19.315999999999999},"min":{"1min":5.8049999999999997,"5min":5.4379999999999997,"15min":4.9199999999999999},"max":{"1min":74.001000000000005,"5min":117.193,"15min":218.78100000000001},"last":11.699999999999999},{"interface":"front","average":{"1min":10.275,"5min":13.943,"15min":15.449},"min":{"1min":5.7229999999999999,"5min":5.1379999999999999,"15min":5.0149999999999997},"max":{"1min":27.335999999999999,"5min":128.50200000000001,"15min":155.27199999999999},"last":7.0999999999999996}]},{"osd":2,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":19.477,"5min":16.728000000000002,"15min":21.635000000000002},"min":{"1min":7.9080000000000004,"5min":6.1870000000000003,"15min":6.1870000000000003},"max":{"1min":67.552999999999997,"5min":122.307,"15min":222.303},"last":7.9889999999999999},{"interface":"front","average":{"1min":16.923999999999999,"5min":18.561,"15min":26.841000000000001},"min":{"1min":7.9580000000000002,"5min":7.9580000000000002,"15min":6.1820000000000004},"max":{"1min":46.755000000000003,"5min":98.781000000000006,"15min":438.19200000000001},"last":17.215}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.385000000000002,"5min":15.81,"15min":17.850000000000001},"min":{"1min":6.4859999999999998,"5min":6.2050000000000001,"15min":6.0179999999999998},"max":{"1min":136.16800000000001,"5min":136.16800000000001,"15min":217.99000000000001},"last":10.569000000000001},{"interface":"front","average":{"1min":30.402000000000001,"5min":20.652999999999999,"15min":21.940000000000001},"min":{"1min":6.3890000000000002,"5min":6.3890000000000002,"15min":6.0010000000000003},"max":{"1min":270.25599999999997,"5min":270.25599999999997,"15min":270.25599999999997},"last":9.8529999999999998}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.241,"5min":21.978000000000002,"15min":23.396000000000001},"min":{"1min":11.669,"5min":8.1539999999999999,"15min":7.5800000000000001},"max":{"1min":69.150000000000006,"5min":168.38800000000001,"15min":190.29599999999999},"last":14.130000000000001},{"interface":"front","average":{"1min":14.085000000000001,"5min":20.285,"15min":22.73},"min":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"max":{"1min":48.207000000000001,"5min":163.41800000000001,"15min":189.59399999999999},"last":18.327000000000002}]},{"osd":5,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":21.626999999999999,"5min":20.792999999999999,"15min":22.007999999999999},"min":{"1min":9.1430000000000007,"5min":8.9939999999999998,"15min":8.9930000000000003},"max":{"1min":68.114000000000004,"5min":133.73400000000001,"15min":138.38800000000001},"last":12.316000000000001},{"interface":"front","average":{"1min":25.091000000000001,"5min":24.527000000000001,"15min":25.061},"min":{"1min":12.673999999999999,"5min":11.9,"15min":10.571},"max":{"1min":68.138999999999996,"5min":150.35499999999999,"15min":152.40199999999999},"last":16.504999999999999}]},{"osd":7,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":17.923999999999999,"5min":18.893999999999998,"15min":23.824999999999999},"min":{"1min":8.8140000000000001,"5min":7.1699999999999999,"15min":6.226},"max":{"1min":64.828999999999994,"5min":98.503,"15min":253.16900000000001},"last":14.802},{"interface":"front","average":{"1min":19.021000000000001,"5min":21.308,"15min":20.143000000000001},"min":{"1min":7.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"max":{"1min":75.846000000000004,"5min":174.071,"15min":221.28399999999999},"last":13.439}]}]},{"osd":5,"up_from":15,"seq":64424509705,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77948,"kb_used_data":6516,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293892,"statfs":{"total":96636764160,"available":96556945408,"internally_reserved":0,"allocated":6672384,"data_stored":3545787,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":17.625,"5min":18.785,"15min":16.260999999999999},"min":{"1min":5.2329999999999997,"5min":3.8359999999999999,"15min":3.8359999999999999},"max":{"1min":85.513999999999996,"5min":220.148,"15min":220.148},"last":11.154},{"interface":"front","average":{"1min":27.861000000000001,"5min":23.675000000000001,"15min":20.949000000000002},"min":{"1min":8.6039999999999992,"5min":5.6059999999999999,"15min":5.6059999999999999},"max":{"1min":145.43000000000001,"5min":177.91,"15min":177.91},"last":15.430999999999999}]},{"osd":1,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":20.715,"5min":16.585000000000001,"15min":20.353999999999999},"min":{"1min":6.3040000000000003,"5min":4.3010000000000002,"15min":4.3010000000000002},"max":{"1min":71.460999999999999,"5min":74.081000000000003,"15min":293.75400000000002},"last":10.182},{"interface":"front","average":{"1min":20.943999999999999,"5min":16.175000000000001,"15min":18.327999999999999},"min":{"1min":5.46,"5min":4.8949999999999996,"15min":4.8949999999999996},"max":{"1min":80.397000000000006,"5min":80.397000000000006,"15min":210.797},"last":13.827}]},{"osd":2,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":31.280000000000001,"5min":23.431999999999999,"15min":25.231000000000002},"min":{"1min":6.2480000000000002,"5min":5.0209999999999999,"15min":5.0209999999999999},"max":{"1min":105.449,"5min":143.273,"15min":519.68799999999999},"last":14.685},{"interface":"front","average":{"1min":23.294,"5min":20.007000000000001,"15min":22.029},"min":{"1min":5.6479999999999997,"5min":5.6479999999999997,"15min":5.226},"max":{"1min":82.042000000000002,"5min":154.083,"15min":218.59899999999999},"last":16.048999999999999}]},{"osd":3,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":41.817,"5min":31.161999999999999,"15min":26.263000000000002},"min":{"1min":9.9900000000000002,"5min":9.9900000000000002,"15min":7.6150000000000002},"max":{"1min":269.52999999999997,"5min":269.52999999999997,"15min":269.52999999999997},"last":11.798999999999999},{"interface":"front","average":{"1min":24.488,"5min":22.629000000000001,"15min":22.704000000000001},"min":{"1min":7.4480000000000004,"5min":6.218,"15min":5.9119999999999999},"max":{"1min":178.78700000000001,"5min":178.78700000000001,"15min":249.797},"last":6.8620000000000001}]},{"osd":4,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":21.725999999999999,"5min":26.437999999999999,"15min":25.25},"min":{"1min":8.1080000000000005,"5min":7.1580000000000004,"15min":6.6100000000000003},"max":{"1min":78.555999999999997,"5min":215.97999999999999,"15min":291.92599999999999},"last":8.1189999999999998},{"interface":"front","average":{"1min":26.097000000000001,"5min":33.600000000000001,"15min":28.585000000000001},"min":{"1min":11.895,"5min":8.0050000000000008,"15min":6.9550000000000001},"max":{"1min":70.290999999999997,"5min":324.31,"15min":324.31},"last":10.798999999999999}]},{"osd":6,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":32.883000000000003,"5min":28.116,"15min":26.379999999999999},"min":{"1min":10.593,"5min":8.6489999999999991,"15min":7.2199999999999998},"max":{"1min":162.964,"5min":211.255,"15min":251.37700000000001},"last":13.17},{"interface":"front","average":{"1min":38.536000000000001,"5min":33.917999999999999,"15min":28.524000000000001},"min":{"1min":9.3640000000000008,"5min":9.2349999999999994,"15min":7.1619999999999999},"max":{"1min":160.42599999999999,"5min":333.29700000000003,"15min":333.29700000000003},"last":17.119}]},{"osd":7,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":36.252000000000002,"5min":33.210000000000001,"15min":27.071999999999999},"min":{"1min":11.131,"5min":10.68,"15min":6.1769999999999996},"max":{"1min":116.877,"5min":258.81,"15min":264.52999999999997},"last":17.988},{"interface":"front","average":{"1min":26.748000000000001,"5min":22.239000000000001,"15min":25.803999999999998},"min":{"1min":7.4580000000000002,"5min":5.8410000000000002,"15min":5.8410000000000002},"max":{"1min":90.570999999999998,"5min":154.86799999999999,"15min":255.625},"last":6.0759999999999996}]}]},{"osd":7,"up_from":14,"seq":60129542410,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59868,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311972,"statfs":{"total":96636764160,"available":96575459328,"internally_reserved":0,"allocated":6635520,"data_stored":3147947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":9.2010000000000005,"5min":13.369999999999999,"15min":14.285},"min":{"1min":4.9580000000000002,"5min":4.0499999999999998,"15min":3.5920000000000001},"max":{"1min":18.266999999999999,"5min":106.79300000000001,"15min":187.85499999999999},"last":11.17},{"interface":"front","average":{"1min":8.7729999999999997,"5min":12.766999999999999,"15min":13.725},"min":{"1min":5.0279999999999996,"5min":4.1020000000000003,"15min":4.1020000000000003},"max":{"1min":15.933,"5min":117.645,"15min":124.342},"last":5.008}]},{"osd":1,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":13.706,"5min":18.475999999999999,"15min":19.167999999999999},"min":{"1min":8.1969999999999992,"5min":5.8979999999999997,"15min":5.7409999999999997},"max":{"1min":33.970999999999997,"5min":103.907,"15min":226.63399999999999},"last":11.821},{"interface":"front","average":{"1min":11.929,"5min":17.565999999999999,"15min":19.605},"min":{"1min":6.5439999999999996,"5min":6.351,"15min":5.859},"max":{"1min":42.719000000000001,"5min":135.41200000000001,"15min":187.24100000000001},"last":10.068}]},{"osd":2,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":17.675000000000001,"5min":18.248999999999999,"15min":20.965},"min":{"1min":7.2869999999999999,"5min":6.2549999999999999,"15min":6.2549999999999999},"max":{"1min":43.064999999999998,"5min":79.168000000000006,"15min":200.285},"last":16.603999999999999},{"interface":"front","average":{"1min":26.209,"5min":21.07,"15min":21.149999999999999},"min":{"1min":6.0679999999999996,"5min":5.5090000000000003,"15min":5.5090000000000003},"max":{"1min":211.315,"5min":211.315,"15min":241.15700000000001},"last":9.0099999999999998}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":25.942,"5min":21.164000000000001,"15min":22.437999999999999},"min":{"1min":10.516,"5min":8.1509999999999998,"15min":6.6849999999999996},"max":{"1min":205.821,"5min":205.821,"15min":262.43900000000002},"last":18.446999999999999},{"interface":"front","average":{"1min":26.780000000000001,"5min":22.042999999999999,"15min":23.443999999999999},"min":{"1min":7.5650000000000004,"5min":6.7460000000000004,"15min":5.9950000000000001},"max":{"1min":225.29900000000001,"5min":225.29900000000001,"15min":330.15300000000002},"last":13.034000000000001}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.153,"5min":21.227,"15min":22.390000000000001},"min":{"1min":9.9459999999999997,"5min":7.218,"15min":7.218},"max":{"1min":22.241,"5min":136.494,"15min":213.072},"last":21.518000000000001},{"interface":"front","average":{"1min":17.73,"5min":22.32,"15min":24.960999999999999},"min":{"1min":8.9540000000000006,"5min":8.9540000000000006,"15min":8.5530000000000008},"max":{"1min":28.087,"5min":104.538,"15min":239.76900000000001},"last":20.449999999999999}]},{"osd":5,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.374000000000001,"5min":18.962,"15min":22.965},"min":{"1min":10.206,"5min":8.2889999999999997,"15min":7.258},"max":{"1min":27.122,"5min":110.45099999999999,"15min":275.08100000000002},"last":13.827},{"interface":"front","average":{"1min":13.379,"5min":15.702999999999999,"15min":18.018000000000001},"min":{"1min":5.3410000000000002,"5min":5.3410000000000002,"15min":5.3410000000000002},"max":{"1min":40.390000000000001,"5min":94.302999999999997,"15min":129.935},"last":17.434000000000001}]},{"osd":6,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":18.283000000000001,"5min":21.891999999999999,"15min":25.209},"min":{"1min":11.725,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":27.867999999999999,"5min":126.001,"15min":263.41399999999999},"last":19.629999999999999},{"interface":"front","average":{"1min":17.962,"5min":19.896000000000001,"15min":23.355},"min":{"1min":11.305999999999999,"5min":8.0229999999999997,"15min":8.0229999999999997},"max":{"1min":56.701000000000001,"5min":111.691,"15min":257.75900000000001},"last":15.065}]}]},{"osd":2,"up_from":14,"seq":60129542410,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82076,"kb_used_data":6484,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289764,"statfs":{"total":96636764160,"available":96552718336,"internally_reserved":0,"allocated":6639616,"data_stored":3147952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":19.824999999999999,"5min":15.673,"15min":19.306999999999999},"min":{"1min":4.6959999999999997,"5min":4.3170000000000002,"15min":4.3170000000000002},"max":{"1min":211.80500000000001,"5min":211.80500000000001,"15min":389.31400000000002},"last":34.222999999999999},{"interface":"front","average":{"1min":18.757999999999999,"5min":20.440999999999999,"15min":19.539000000000001},"min":{"1min":5.9269999999999996,"5min":5.3040000000000003,"15min":4.4589999999999996},"max":{"1min":213.93600000000001,"5min":295.87,"15min":295.87},"last":14.84}]},{"osd":1,"last update":"Mon Oct 30 20:45:48 2023","interfaces":[{"interface":"back","average":{"1min":14.757,"5min":16.274999999999999,"15min":22.295999999999999},"min":{"1min":6.4720000000000004,"5min":5.5060000000000002,"15min":5.2649999999999997},"max":{"1min":36.689,"5min":74.575000000000003,"15min":245.84999999999999},"last":8.1929999999999996},{"interface":"front","average":{"1min":17.843,"5min":15.342000000000001,"15min":20.946999999999999},"min":{"1min":7.9459999999999997,"5min":5.8399999999999999,"15min":5.3330000000000002},"max":{"1min":66.947999999999993,"5min":72.662999999999997,"15min":317.20699999999999},"last":9.3889999999999993}]},{"osd":3,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.094000000000001,"5min":17.843,"15min":22.158000000000001},"min":{"1min":6.6040000000000001,"5min":6.1459999999999999,"15min":5.2619999999999996},"max":{"1min":58.442999999999998,"5min":253.02799999999999,"15min":311.03100000000001},"last":7.5970000000000004},{"interface":"front","average":{"1min":20.954000000000001,"5min":21.210999999999999,"15min":28.140999999999998},"min":{"1min":10.403,"5min":7.4969999999999999,"15min":6.2039999999999997},"max":{"1min":86.935000000000002,"5min":145.066,"15min":703},"last":12.502000000000001}]},{"osd":4,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":14.718,"5min":18.241,"15min":26.344999999999999},"min":{"1min":7.4029999999999996,"5min":6.4809999999999999,"15min":6.3460000000000001},"max":{"1min":43.838999999999999,"5min":123.06399999999999,"15min":709.85900000000004},"last":10.69},{"interface":"front","average":{"1min":16.712,"5min":19.233000000000001,"15min":22.349},"min":{"1min":6.2359999999999998,"5min":6.2359999999999998,"15min":6.2359999999999998},"max":{"1min":65.897000000000006,"5min":118.82599999999999,"15min":182.947},"last":11.893000000000001}]},{"osd":5,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":18.765000000000001,"5min":18.805,"15min":29.754000000000001},"min":{"1min":8.3539999999999992,"5min":7.2279999999999998,"15min":6.8460000000000001},"max":{"1min":63.963000000000001,"5min":116.352,"15min":874.22799999999995},"last":13.478},{"interface":"front","average":{"1min":17.289999999999999,"5min":18.369,"15min":24.251999999999999},"min":{"1min":6.9500000000000002,"5min":6.9500000000000002,"15min":6.9500000000000002},"max":{"1min":41.155000000000001,"5min":73.218999999999994,"15min":222.76400000000001},"last":16.359999999999999}]},{"osd":6,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":15.76,"5min":23.898,"15min":25.614999999999998},"min":{"1min":8.5389999999999997,"5min":8.5389999999999997,"15min":7.3049999999999997},"max":{"1min":37.296999999999997,"5min":196.654,"15min":238.84800000000001},"last":14.173999999999999},{"interface":"front","average":{"1min":18.132000000000001,"5min":23.280000000000001,"15min":29.577000000000002},"min":{"1min":8.3770000000000007,"5min":8.0670000000000002,"15min":7.577},"max":{"1min":37.128999999999998,"5min":161.30799999999999,"15min":266.48599999999999},"last":19.367000000000001}]},{"osd":7,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":19.760000000000002,"5min":20.834,"15min":26.890000000000001},"min":{"1min":9.7300000000000004,"5min":9.7300000000000004,"15min":9.7300000000000004},"max":{"1min":39.747,"5min":133.429,"15min":354.904},"last":18.393999999999998},{"interface":"front","average":{"1min":19.949999999999999,"5min":21.204000000000001,"15min":29.919},"min":{"1min":9.1600000000000001,"5min":7.3440000000000003,"15min":7.3440000000000003},"max":{"1min":75.688999999999993,"5min":153.803,"15min":865.25800000000004},"last":17.376000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575116,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82076,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289764,"statfs":{"total":96636764160,"available":96552718336,"internally_reserved":0,"allocated":6635520,"data_stored":3147947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":12.099,"5min":10.295,"15min":15.339},"min":{"1min":4.5940000000000003,"5min":3.8839999999999999,"15min":3.839},"max":{"1min":73.688000000000002,"5min":118.02500000000001,"15min":224.95099999999999},"last":6.3399999999999999},{"interface":"front","average":{"1min":14.191000000000001,"5min":11.253,"15min":15.319000000000001},"min":{"1min":4.407,"5min":4.407,"15min":4.0229999999999997},"max":{"1min":70.055999999999997,"5min":70.055999999999997,"15min":227.71600000000001},"last":6.968}]},{"osd":1,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":19.260999999999999,"5min":15.923,"15min":20.707999999999998},"min":{"1min":6.3550000000000004,"5min":4.7960000000000003,"15min":4.7960000000000003},"max":{"1min":92.763000000000005,"5min":118.747,"15min":383.57499999999999},"last":13.497},{"interface":"front","average":{"1min":20.491,"5min":19.440000000000001,"15min":22.885999999999999},"min":{"1min":8.3000000000000007,"5min":7.8970000000000002,"15min":7.5},"max":{"1min":59.844999999999999,"5min":110.129,"15min":228.34999999999999},"last":14.859999999999999}]},{"osd":2,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":17.518000000000001,"5min":13.864000000000001,"15min":21.329000000000001},"min":{"1min":8.2390000000000008,"5min":4.6420000000000003,"15min":4.6420000000000003},"max":{"1min":54.417999999999999,"5min":82.471999999999994,"15min":347.30799999999999},"last":11.335000000000001},{"interface":"front","average":{"1min":23.317,"5min":16.451000000000001,"15min":21.233000000000001},"min":{"1min":6.1369999999999996,"5min":6.1369999999999996,"15min":6.0090000000000003},"max":{"1min":121.45999999999999,"5min":121.45999999999999,"15min":183.03100000000001},"last":7.9400000000000004}]},{"osd":3,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":16.106000000000002,"5min":17.736000000000001,"15min":24.536000000000001},"min":{"1min":6.4779999999999998,"5min":6.4779999999999998,"15min":6.2930000000000001},"max":{"1min":70.343999999999994,"5min":174.733,"15min":270.19400000000002},"last":10.682},{"interface":"front","average":{"1min":18.103999999999999,"5min":17.949999999999999,"15min":24.756},"min":{"1min":6.0220000000000002,"5min":6.0220000000000002,"15min":6.0220000000000002},"max":{"1min":63.222000000000001,"5min":177.87,"15min":339.79700000000003},"last":15.968}]},{"osd":5,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":20.681000000000001,"5min":20.640999999999998,"15min":24.623999999999999},"min":{"1min":6.5839999999999996,"5min":6.5839999999999996,"15min":6.5839999999999996},"max":{"1min":158.541,"5min":158.541,"15min":253.80199999999999},"last":16.797000000000001},{"interface":"front","average":{"1min":17.707000000000001,"5min":16.873000000000001,"15min":22.225000000000001},"min":{"1min":6.1029999999999998,"5min":6.1029999999999998,"15min":6.1029999999999998},"max":{"1min":147.54400000000001,"5min":147.54400000000001,"15min":271.572},"last":9.3109999999999999}]},{"osd":6,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":26.273,"5min":18.638999999999999,"15min":24.821999999999999},"min":{"1min":7.0999999999999996,"5min":6.4210000000000003,"15min":6.4210000000000003},"max":{"1min":245.447,"5min":245.447,"15min":254.44399999999999},"last":12.161},{"interface":"front","average":{"1min":18.928999999999998,"5min":19.975000000000001,"15min":28.715},"min":{"1min":10.25,"5min":9.0419999999999998,"15min":8.1639999999999997},"max":{"1min":72.599000000000004,"5min":74.463999999999999,"15min":291.62599999999998},"last":17.664999999999999}]},{"osd":7,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":14.513,"5min":16.260000000000002,"15min":22.474},"min":{"1min":8.1890000000000001,"5min":8.1890000000000001,"15min":6.2389999999999999},"max":{"1min":43.911999999999999,"5min":76.325000000000003,"15min":262.82100000000003},"last":10.087999999999999},{"interface":"front","average":{"1min":19.515000000000001,"5min":19.870999999999999,"15min":25.632000000000001},"min":{"1min":9.8350000000000009,"5min":9.8350000000000009,"15min":9.8350000000000009},"max":{"1min":61.091000000000001,"5min":61.091000000000001,"15min":244.37799999999999},"last":14.159000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T20:46:01.868 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.868 INFO:tasks.ceph:pgid 10.17 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.869 INFO:tasks.ceph:pgid 10.14 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.869 INFO:tasks.ceph:pgid 10.15 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.869 INFO:tasks.ceph:pgid 10.12 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.869 INFO:tasks.ceph:pgid 10.13 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.870 INFO:tasks.ceph:pgid 10.10 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.870 INFO:tasks.ceph:pgid 10.11 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.870 INFO:tasks.ceph:pgid 10.1e last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.871 INFO:tasks.ceph:pgid 10.1f last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.871 INFO:tasks.ceph:pgid 10.1c last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.871 INFO:tasks.ceph:pgid 10.1d last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.871 INFO:tasks.ceph:pgid 10.f last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.872 INFO:tasks.ceph:pgid 10.e last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.872 INFO:tasks.ceph:pgid 10.d last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.872 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T20:24:02.536651+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=24, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.872 INFO:tasks.ceph:pgid 10.c last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.873 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-30T20:24:02.536651+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=24, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.873 INFO:tasks.ceph:pgid 10.9 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.873 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-30T20:24:02.536651+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=24, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.874 INFO:tasks.ceph:pgid 10.8 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.874 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-30T20:24:02.536651+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=24, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.874 INFO:tasks.ceph:pgid 10.b last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.874 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-30T20:23:54.534733+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=23, tm_sec=54, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.875 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.875 INFO:tasks.ceph:pgid 10.3 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.875 INFO:tasks.ceph:pgid 10.2 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.875 INFO:tasks.ceph:pgid 10.5 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.876 INFO:tasks.ceph:pgid 10.4 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.876 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.876 INFO:tasks.ceph:pgid 10.6 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.876 INFO:tasks.ceph:pgid 10.19 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.877 INFO:tasks.ceph:pgid 10.18 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.877 INFO:tasks.ceph:pgid 10.1b last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.877 INFO:tasks.ceph:pgid 10.1a last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:01.877 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T20:46:10.372 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:32.431 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:10.374 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:32.433 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:10.390 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:32.448 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:10.419 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:32.478 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:10.422 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:32.449 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:10.423 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:32.482 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:10.423 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:32.451 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:10.441 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:32.500 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:10.442 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:32.469 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:15.343 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:37.402 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:15.387 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:37.415 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:15.389 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:37.447 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:15.391 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:37.418 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:15.402 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:37.461 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:15.402 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:37.430 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:15.440 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:37.499 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:15.442 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:37.501 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:15.458 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:37.517 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:21.879 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T20:46:21.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.976+0000 7f3caef1d640 10 monclient: get_monmap_and_config 2023-10-30T20:46:21.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.976+0000 7f3caef1d640 10 monclient: build_initial_monmap 2023-10-30T20:46:21.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.976+0000 7f3caef1d640 10 monclient: monmap: 2023-10-30T20:46:21.979 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:46:21.979 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:21.980 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:46:21.977738+0000 2023-10-30T20:46:21.980 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:46:21.977738+0000 2023-10-30T20:46:21.980 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:46:21.980 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:21.981 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:46:21.981 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:46:21.981 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:46:21.982 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:46:21.982 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:46:21.982 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:21.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.977+0000 7f3caef1d640 1 Processor -- start 2023-10-30T20:46:21.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.977+0000 7f3caef1d640 1 -- start start 2023-10-30T20:46:21.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.977+0000 7f3caef1d640 10 monclient: init 2023-10-30T20:46:21.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caef1d640 10 monclient: _reopen_session rank -1 2023-10-30T20:46:21.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caef1d640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-10-30T20:46:21.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caef1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3ca8142e40 addr v1:172.21.15.183:6789/0 2023-10-30T20:46:21.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caef1d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3ca80ac3e0 0x7f3ca806ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caef1d640 10 monclient(hunting): picked mon.noname-a con 0x7f3ca80ac3e0 addr v2:172.21.15.69:3300/0 2023-10-30T20:46:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caef1d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca8070480 0x7f3ca8149f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caef1d640 10 monclient(hunting): picked mon.noname-b con 0x7f3ca8070480 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:46:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caef1d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca8143ad0 con 0x7f3ca8142e40 2023-10-30T20:46:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caef1d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:21.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caef1d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca8067b90 con 0x7f3ca80ac3e0 2023-10-30T20:46:21.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caef1d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:21.993 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caef1d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca8071a90 con 0x7f3ca8070480 2023-10-30T20:46:21.993 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caef1d640 10 monclient(hunting): _renew_subs 2023-10-30T20:46:21.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caef1d640 10 monclient(hunting): authenticate will time out at 2905.266911s 2023-10-30T20:46:21.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3cad4ba640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca8070480 0x7f3ca8149f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:21.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caccb9640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f3ca8142e40 0x7f3ca80abcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.69:47068/0 (socket says 172.21.15.69:47068) 2023-10-30T20:46:21.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3caccb9640 1 -- 172.21.15.69:0/2703440497 learned_addr learned my addr 172.21.15.69:0/2703440497 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:46:21.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.978+0000 7f3ca7fff640 1 --2- 172.21.15.69:0/2703440497 >> v2:172.21.15.69:3300/0 conn(0x7f3ca80ac3e0 0x7f3ca806ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:21.996 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.979+0000 7f3cad4ba640 10 monclient(hunting): get_auth_request con 0x7f3ca8070480 auth_method 0 2023-10-30T20:46:21.996 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.979+0000 7f3cad4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:21.997 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.979+0000 7f3cad4ba640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:21.997 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.979+0000 7f3cad4ba640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:21.997 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.979+0000 7f3ca7fff640 10 monclient(hunting): get_auth_request con 0x7f3ca80ac3e0 auth_method 0 2023-10-30T20:46:21.998 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.979+0000 7f3ca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:21.998 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.979+0000 7f3ca7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:21.999 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.979+0000 7f3ca7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:21.999 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.979+0000 7f3cad4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:22.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.979+0000 7f3cad4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:22.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.979+0000 7f3cad4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:22.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.979+0000 7f3ca7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:22.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.979+0000 7f3ca7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:22.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.979+0000 7f3ca7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:22.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3ca77fe640 1 -- 172.21.15.69:0/2703440497 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545250024 0 0) 0x7f3ca8143ad0 con 0x7f3ca8142e40 2023-10-30T20:46:22.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3ca77fe640 10 monclient(hunting): _negotiate 2023-10-30T20:46:22.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3ca77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:22.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3ca77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:22.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3ca77fe640 10 monclient(hunting): my global_id is 21319 2023-10-30T20:46:22.004 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3ca77fe640 1 -- 172.21.15.69:0/2703440497 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c98003520 con 0x7f3ca8142e40 2023-10-30T20:46:22.004 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3cad4ba640 10 monclient(hunting): handle_auth_done global_id 18356 payload 274 2023-10-30T20:46:22.005 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3cad4ba640 1 -- 172.21.15.69:0/2703440497 >> v2:172.21.15.69:3300/0 conn(0x7f3ca80ac3e0 msgr2=0x7f3ca806ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:46:22.005 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3cad4ba640 1 --2- 172.21.15.69:0/2703440497 >> v2:172.21.15.69:3300/0 conn(0x7f3ca80ac3e0 0x7f3ca806ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:22.005 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3cad4ba640 1 -- 172.21.15.69:0/2703440497 >> v1:172.21.15.183:6789/0 conn(0x7f3ca8142e40 legacy=0x7f3ca80abcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:22.006 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3cad4ba640 10 monclient: _finish_hunting 0 2023-10-30T20:46:22.006 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3cad4ba640 1 monclient: found mon.noname-b 2023-10-30T20:46:22.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3cad4ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:46:22.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3cad4ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:46:22.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.981+0000 7f3cad4ba640 1 -- 172.21.15.69:0/2703440497 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca80a8470 con 0x7f3ca8070480 2023-10-30T20:46:22.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.982+0000 7f3cad4ba640 1 --2- 172.21.15.69:0/2703440497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca8070480 0x7f3ca8149f60 secure :-1 s=READY pgs=3220 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009d90 tx=0x7f3ca002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6e66865a608d63a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:22.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3ca77fe640 1 -- 172.21.15.69:0/2703440497 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca002cc50 con 0x7f3ca8070480 2023-10-30T20:46:22.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3ca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:22.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3ca77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:46:22.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3ca77fe640 10 monclient: dump: 2023-10-30T20:46:22.019 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:22.019 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:22.019 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:22.019 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:22.020 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:22.020 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:22.020 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:22.021 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:22.021 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:22.021 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:22.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3ca77fe640 10 monclient: _finish_auth 0 2023-10-30T20:46:22.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3ca77fe640 10 monclient: _check_auth_tickets 2023-10-30T20:46:22.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3ca77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:46:22.023 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3ca77fe640 1 -- 172.21.15.69:0/2703440497 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca002cdc0 con 0x7f3ca8070480 2023-10-30T20:46:22.023 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3ca77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:46:22.023 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3ca77fe640 1 -- 172.21.15.69:0/2703440497 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0034660 con 0x7f3ca8070480 2023-10-30T20:46:22.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3caef1d640 5 monclient: authenticate success, global_id 18356 2023-10-30T20:46:22.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3caef1d640 10 monclient: get_monmap_and_config success 2023-10-30T20:46:22.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3caef1d640 4 set_mon_vals no callback set 2023-10-30T20:46:22.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3ca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:22.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3ca77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:46:22.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.986+0000 7f3ca77fe640 10 monclient: dump: 2023-10-30T20:46:22.026 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:22.026 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:22.026 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:22.027 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:22.027 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:22.027 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:22.029 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:22.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.987+0000 7f3caef1d640 10 monclient: shutdown 2023-10-30T20:46:22.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.987+0000 7f3caef1d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:46:22.030 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.987+0000 7f3caef1d640 1 -- 172.21.15.69:0/2703440497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca8070480 msgr2=0x7f3ca8149f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:22.030 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.987+0000 7f3caef1d640 1 --2- 172.21.15.69:0/2703440497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca8070480 0x7f3ca8149f60 secure :-1 s=READY pgs=3220 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009d90 tx=0x7f3ca002bea0 comp rx=0 tx=0).stop 2023-10-30T20:46:22.030 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.987+0000 7f3caef1d640 1 -- 172.21.15.69:0/2703440497 shutdown_connections 2023-10-30T20:46:22.031 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.987+0000 7f3caef1d640 1 --2- 172.21.15.69:0/2703440497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca8070480 0x7f3ca8149f60 unknown :-1 s=CLOSED pgs=3220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:22.031 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.987+0000 7f3caef1d640 1 --2- 172.21.15.69:0/2703440497 >> v2:172.21.15.69:3300/0 conn(0x7f3ca80ac3e0 0x7f3ca806ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:22.031 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.987+0000 7f3caef1d640 1 -- 172.21.15.69:0/2703440497 >> 172.21.15.69:0/2703440497 conn(0x7f3ca80afb80 msgr2=0x7f3ca80aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:46:22.032 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.987+0000 7f3caef1d640 1 -- 172.21.15.69:0/2703440497 shutdown_connections 2023-10-30T20:46:22.032 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.987+0000 7f3caef1d640 1 -- 172.21.15.69:0/2703440497 wait complete. 2023-10-30T20:46:22.032 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.988+0000 7f3caef1d640 10 monclient: build_initial_monmap 2023-10-30T20:46:22.033 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.988+0000 7f3caef1d640 10 monclient: monmap: 2023-10-30T20:46:22.033 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:46:22.033 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:46:22.034 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:46:22.034 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:46:22.034 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:46:22.035 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:46:22.035 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:46:22.035 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:46:22.036 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:46:22.036 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:22.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.988+0000 7f3caef1d640 1 librados: starting msgr at 2023-10-30T20:46:22.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.988+0000 7f3caef1d640 1 librados: starting objecter 2023-10-30T20:46:22.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.988+0000 7f3caef1d640 1 Processor -- start 2023-10-30T20:46:22.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.988+0000 7f3caef1d640 1 -- start start 2023-10-30T20:46:22.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.988+0000 7f3caef1d640 1 librados: setting wanted keys 2023-10-30T20:46:22.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.988+0000 7f3caef1d640 1 librados: calling monclient init 2023-10-30T20:46:22.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.988+0000 7f3caef1d640 10 monclient: init 2023-10-30T20:46:22.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 10 monclient: _reopen_session rank -1 2023-10-30T20:46:22.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T20:46:22.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca8070480 0x7f3ca813e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:22.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 10 monclient(hunting): picked mon.noname-c con 0x7f3ca8070480 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:46:22.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ca80ac3e0 0x7f3ca806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:22.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 10 monclient(hunting): picked mon.noname-b con 0x7f3ca80ac3e0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:46:22.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ca8142e40 0x7f3ca806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:22.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 10 monclient(hunting): picked mon.noname-a con 0x7f3ca8142e40 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:46:22.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:22.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca80af4e0 con 0x7f3ca8142e40 2023-10-30T20:46:22.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:22.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca813edd0 con 0x7f3ca80ac3e0 2023-10-30T20:46:22.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:22.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca813e630 con 0x7f3ca8070480 2023-10-30T20:46:22.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 10 monclient(hunting): _renew_subs 2023-10-30T20:46:22.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caef1d640 10 monclient(hunting): authenticate will time out at 2905.278172s 2023-10-30T20:46:22.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3caccb9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca8070480 0x7f3ca813e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:22.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.989+0000 7f3cad4ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ca8142e40 0x7f3ca806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:22.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3caccb9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca8070480 0x7f3ca813e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41456/0 (socket says 172.21.15.69:41456) 2023-10-30T20:46:22.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3caccb9640 1 -- 172.21.15.69:0/2410708998 learned_addr learned my addr 172.21.15.69:0/2410708998 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:46:22.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3caccb9640 10 monclient(hunting): get_auth_request con 0x7f3ca8070480 auth_method 0 2023-10-30T20:46:22.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3caccb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:22.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3caccb9640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:22.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3caccb9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:22.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3cad4ba640 10 monclient(hunting): get_auth_request con 0x7f3ca8142e40 auth_method 0 2023-10-30T20:46:22.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3cad4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:22.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3cad4ba640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:22.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3cad4ba640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:22.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3ca7fff640 1 --2- 172.21.15.69:0/2410708998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ca80ac3e0 0x7f3ca806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:22.050 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3ca7fff640 10 monclient(hunting): get_auth_request con 0x7f3ca80ac3e0 auth_method 0 2023-10-30T20:46:22.050 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3ca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:22.051 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3ca7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:22.051 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3ca7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:22.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3caccb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:22.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3caccb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:22.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3caccb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:22.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3cad4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:22.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3cad4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:22.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.990+0000 7f3cad4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:22.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.991+0000 7f3ca7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:22.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.991+0000 7f3ca7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:22.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.991+0000 7f3ca7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:22.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.993+0000 7f3caccb9640 10 monclient(hunting): handle_auth_done global_id 18359 payload 824 2023-10-30T20:46:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.993+0000 7f3caccb9640 1 -- 172.21.15.69:0/2410708998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ca80ac3e0 msgr2=0x7f3ca806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.993+0000 7f3caccb9640 1 --2- 172.21.15.69:0/2410708998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ca80ac3e0 0x7f3ca806d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.993+0000 7f3caccb9640 1 -- 172.21.15.69:0/2410708998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ca8142e40 msgr2=0x7f3ca806d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.993+0000 7f3caccb9640 1 --2- 172.21.15.69:0/2410708998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ca8142e40 0x7f3ca806d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:22.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.993+0000 7f3caccb9640 10 monclient: _finish_hunting 0 2023-10-30T20:46:22.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.993+0000 7f3caccb9640 1 monclient: found mon.noname-c 2023-10-30T20:46:22.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.993+0000 7f3caccb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:46:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.993+0000 7f3caccb9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:46:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.993+0000 7f3caccb9640 1 -- 172.21.15.69:0/2410708998 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca813f260 con 0x7f3ca8070480 2023-10-30T20:46:22.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.995+0000 7f3caccb9640 1 --2- 172.21.15.69:0/2410708998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca8070480 0x7f3ca813e060 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c00c390 tx=0x7f3c9c00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:22.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.995+0000 7f3ca57fa640 10 client.?.objecter ms_handle_connect 0x7f3ca8070480 2023-10-30T20:46:22.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.995+0000 7f3ca57fa640 10 client.?.objecter resend_mon_ops 2023-10-30T20:46:22.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.998+0000 7f3ca57fa640 1 -- 172.21.15.69:0/2410708998 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c005ce0 con 0x7f3ca8070480 2023-10-30T20:46:22.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.998+0000 7f3ca57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:22.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.998+0000 7f3ca57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:46:22.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.998+0000 7f3ca57fa640 10 monclient: dump: 2023-10-30T20:46:22.062 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:22.063 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:22.063 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:22.063 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:22.064 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:22.064 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:22.064 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:22.065 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:22.065 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:22.065 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:22.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3ca57fa640 10 monclient: _finish_auth 0 2023-10-30T20:46:22.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3ca57fa640 10 monclient: _check_auth_tickets 2023-10-30T20:46:22.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3ca57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:46:22.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3ca57fa640 1 -- 172.21.15.69:0/2410708998 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9c00e040 con 0x7f3ca8070480 2023-10-30T20:46:22.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3caef1d640 5 monclient: authenticate success, global_id 18359 2023-10-30T20:46:22.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3caef1d640 10 monclient: _renew_subs 2023-10-30T20:46:22.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3caef1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:46:22.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3caef1d640 1 -- 172.21.15.69:0/2410708998 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca806db30 con 0x7f3ca8070480 2023-10-30T20:46:22.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3ca57fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:46:22.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3ca57fa640 1 -- 172.21.15.69:0/2410708998 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c0072f0 con 0x7f3ca8070480 2023-10-30T20:46:22.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3ca57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:22.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3ca57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:46:22.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3ca57fa640 10 monclient: dump: 2023-10-30T20:46:22.070 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:22.070 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:22.070 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:22.071 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:22.071 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:22.071 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:22.072 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:22.072 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:22.072 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:22.073 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:22.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3ca6ffd640 4 set_mon_vals no callback set 2023-10-30T20:46:22.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3caef1d640 10 client.18359.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:46:22.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3caef1d640 10 monclient: _renew_subs 2023-10-30T20:46:22.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3caef1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:46:22.074 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3caef1d640 1 -- 172.21.15.69:0/2410708998 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca806ef20 con 0x7f3ca8070480 2023-10-30T20:46:22.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3caef1d640 1 librados: init done 2023-10-30T20:46:22.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3caef1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:22.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3caef1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:22.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3caef1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:46:22.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:21.999+0000 7f3caef1d640 1 -- 172.21.15.69:0/2410708998 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca804e9b0 con 0x7f3ca8070480 2023-10-30T20:46:22.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.017+0000 7f3ca57fa640 1 -- 172.21.15.69:0/2410708998 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f3c9c007460 con 0x7f3ca8070480 2023-10-30T20:46:22.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.017+0000 7f3ca57fa640 1 --2- 172.21.15.69:0/2410708998 >> v2:172.21.15.69:6800/1876307353 conn(0x7f3c8c06c7c0 0x7f3c8c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:22.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.017+0000 7f3ca7fff640 1 --2- 172.21.15.69:0/2410708998 >> v2:172.21.15.69:6800/1876307353 conn(0x7f3c8c06c7c0 0x7f3c8c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:22.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.017+0000 7f3ca7fff640 10 monclient: get_auth_request con 0x7f3c8c06c7c0 auth_method 0 2023-10-30T20:46:22.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.018+0000 7f3ca57fa640 10 client.18359.objecter ms_handle_connect 0x7f3c8c06c7c0 2023-10-30T20:46:22.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.018+0000 7f3ca7fff640 1 --2- 172.21.15.69:0/2410708998 >> v2:172.21.15.69:6800/1876307353 conn(0x7f3c8c06c7c0 0x7f3c8c06ec80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8140830 tx=0x7f3c94009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:22.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.019+0000 7f3ca57fa640 1 -- 172.21.15.69:0/2410708998 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f3c9c007710 con 0x7f3ca8070480 2023-10-30T20:46:22.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.019+0000 7f3ca57fa640 10 client.18359.objecter ms_dispatch 0x7f3ca8000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:46:22.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.019+0000 7f3ca57fa640 3 client.18359.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:46:22.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.019+0000 7f3ca57fa640 3 client.18359.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:46:22.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.019+0000 7f3ca57fa640 20 client.18359.objecter dump_active .. 0 homeless 2023-10-30T20:46:22.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.254+0000 7f3ca57fa640 1 -- 172.21.15.69:0/2410708998 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3c9c057800 con 0x7f3ca8070480 2023-10-30T20:46:22.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.254+0000 7f3ca57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:22.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.254+0000 7f3ca57fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:46:22.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.424+0000 7f3caef1d640 1 -- 172.21.15.69:0/2410708998 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ca813d190 con 0x7f3c8c06c7c0 2023-10-30T20:46:22.430 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.428+0000 7f3ca57fa640 1 -- 172.21.15.69:0/2410708998 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152099 (secure 0 0 0) 0x7f3c94011a30 con 0x7f3c8c06c7c0 2023-10-30T20:46:22.432 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:46:22.433 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-10-30T20:46:22.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 10 librados: watch_flush enter 2023-10-30T20:46:22.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 10 librados: watch_flush exit 2023-10-30T20:46:22.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 20 client.18359.objecter shutdown clearing up homeless session... 2023-10-30T20:46:22.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 10 client.18359.objecter successfully canceled tick 2023-10-30T20:46:22.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 1 -- 172.21.15.69:0/2410708998 >> v2:172.21.15.69:6800/1876307353 conn(0x7f3c8c06c7c0 msgr2=0x7f3c8c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:22.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 1 --2- 172.21.15.69:0/2410708998 >> v2:172.21.15.69:6800/1876307353 conn(0x7f3c8c06c7c0 0x7f3c8c06ec80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8140830 tx=0x7f3c94009190 comp rx=0 tx=0).stop 2023-10-30T20:46:22.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 10 monclient: shutdown 2023-10-30T20:46:22.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:46:22.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 1 -- 172.21.15.69:0/2410708998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca8070480 msgr2=0x7f3ca813e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:22.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 1 --2- 172.21.15.69:0/2410708998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca8070480 0x7f3ca813e060 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c00c390 tx=0x7f3c9c00c850 comp rx=0 tx=0).stop 2023-10-30T20:46:22.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 1 -- 172.21.15.69:0/2410708998 shutdown_connections 2023-10-30T20:46:22.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 1 --2- 172.21.15.69:0/2410708998 >> v2:172.21.15.69:6800/1876307353 conn(0x7f3c8c06c7c0 0x7f3c8c06ec80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:22.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 1 --2- 172.21.15.69:0/2410708998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ca8142e40 0x7f3ca806d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:22.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 1 --2- 172.21.15.69:0/2410708998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ca80ac3e0 0x7f3ca806d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:22.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 1 --2- 172.21.15.69:0/2410708998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca8070480 0x7f3ca813e060 unknown :-1 s=CLOSED pgs=3221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:22.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 1 -- 172.21.15.69:0/2410708998 >> 172.21.15.69:0/2410708998 conn(0x7f3ca80afb80 msgr2=0x7f3ca80aff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:46:22.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 1 -- 172.21.15.69:0/2410708998 shutdown_connections 2023-10-30T20:46:22.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 1 -- 172.21.15.69:0/2410708998 wait complete. 2023-10-30T20:46:22.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:22.431+0000 7f3caef1d640 1 librados: shutdown 2023-10-30T20:46:22.446 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1582,"stamp":"2023-10-30T20:46:20.508715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":545616,"kb_used_data":51920,"kb_used_omap":12,"kb_used_meta":493619,"kb_avail":754429104,"statfs":{"total":773094113280,"available":772535402496,"internally_reserved":0,"allocated":53166080,"data_stored":25979266,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12745,"internal_metadata":505466423},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002354"},"pg_stats":[{"pgid":"10.16","version":"0'0","reported_seq":856,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:57.478854+0000","last_change":"2023-10-30T20:45:57.479913+0000","last_active":"2023-10-30T20:45:57.478854+0000","last_peered":"2023-10-30T20:45:57.478854+0000","last_clean":"2023-10-30T20:45:57.478854+0000","last_became_active":"2023-10-30T20:25:23.962078+0000","last_became_peered":"2023-10-30T20:25:23.962078+0000","last_unstale":"2023-10-30T20:45:57.478854+0000","last_undegraded":"2023-10-30T20:45:57.478854+0000","last_fullsized":"2023-10-30T20:45:57.478854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:57.477507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:57.477507+0000","last_clean_scrub_stamp":"2023-10-30T20:45:57.477507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:20:59.025555+0000","scrub_duration":0.036447980999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":872,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.725841+0000","last_change":"2023-10-30T20:25:24.870906+0000","last_active":"2023-10-30T20:45:55.725841+0000","last_peered":"2023-10-30T20:45:55.725841+0000","last_clean":"2023-10-30T20:45:55.725841+0000","last_became_active":"2023-10-30T20:25:24.861824+0000","last_became_peered":"2023-10-30T20:25:24.861824+0000","last_unstale":"2023-10-30T20:45:55.725841+0000","last_undegraded":"2023-10-30T20:45:55.725841+0000","last_fullsized":"2023-10-30T20:45:55.725841+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":861,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:04.227657+0000","last_change":"2023-10-30T20:46:04.228990+0000","last_active":"2023-10-30T20:46:04.227657+0000","last_peered":"2023-10-30T20:46:04.227657+0000","last_clean":"2023-10-30T20:46:04.227657+0000","last_became_active":"2023-10-30T20:25:36.951760+0000","last_became_peered":"2023-10-30T20:25:36.951760+0000","last_unstale":"2023-10-30T20:46:04.227657+0000","last_undegraded":"2023-10-30T20:46:04.227657+0000","last_fullsized":"2023-10-30T20:46:04.227657+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:04.226007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:04.226007+0000","last_clean_scrub_stamp":"2023-10-30T20:46:04.226007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:49:58.849033+0000","scrub_duration":0.052304344000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":878,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:12.909228+0000","last_change":"2023-10-30T20:46:12.910204+0000","last_active":"2023-10-30T20:46:12.909228+0000","last_peered":"2023-10-30T20:46:12.909228+0000","last_clean":"2023-10-30T20:46:12.909228+0000","last_became_active":"2023-10-30T20:25:23.356998+0000","last_became_peered":"2023-10-30T20:25:23.356998+0000","last_unstale":"2023-10-30T20:46:12.909228+0000","last_undegraded":"2023-10-30T20:46:12.909228+0000","last_fullsized":"2023-10-30T20:46:12.909228+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:12.907859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:12.907859+0000","last_clean_scrub_stamp":"2023-10-30T20:46:12.907859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:12:47.708802+0000","scrub_duration":0.042919487999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":889,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:16.846325+0000","last_change":"2023-10-30T20:46:16.847330+0000","last_active":"2023-10-30T20:46:16.846325+0000","last_peered":"2023-10-30T20:46:16.846325+0000","last_clean":"2023-10-30T20:46:16.846325+0000","last_became_active":"2023-10-30T20:25:25.477142+0000","last_became_peered":"2023-10-30T20:25:25.477142+0000","last_unstale":"2023-10-30T20:46:16.846325+0000","last_undegraded":"2023-10-30T20:46:16.846325+0000","last_fullsized":"2023-10-30T20:46:16.846325+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:16.844987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:16.844987+0000","last_clean_scrub_stamp":"2023-10-30T20:46:16.844987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:14:32.688895+0000","scrub_duration":0.042022205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":861,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:02.184943+0000","last_change":"2023-10-30T20:46:02.185985+0000","last_active":"2023-10-30T20:46:02.184943+0000","last_peered":"2023-10-30T20:46:02.184943+0000","last_clean":"2023-10-30T20:46:02.184943+0000","last_became_active":"2023-10-30T20:25:23.840599+0000","last_became_peered":"2023-10-30T20:25:23.840599+0000","last_unstale":"2023-10-30T20:46:02.184943+0000","last_undegraded":"2023-10-30T20:46:02.184943+0000","last_fullsized":"2023-10-30T20:46:02.184943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:02.183565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:02.183565+0000","last_clean_scrub_stamp":"2023-10-30T20:46:02.183565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:23:34.216637+0000","scrub_duration":0.065721632000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":890,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:01.765247+0000","last_change":"2023-10-30T20:46:01.765247+0000","last_active":"2023-10-30T20:46:01.765247+0000","last_peered":"2023-10-30T20:46:01.765247+0000","last_clean":"2023-10-30T20:46:01.765247+0000","last_became_active":"2023-10-30T20:25:28.691162+0000","last_became_peered":"2023-10-30T20:25:28.691162+0000","last_unstale":"2023-10-30T20:46:01.765247+0000","last_undegraded":"2023-10-30T20:46:01.765247+0000","last_fullsized":"2023-10-30T20:46:01.765247+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:01.759908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:01.759908+0000","last_clean_scrub_stamp":"2023-10-30T20:46:01.759908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:58.795323+0000","scrub_duration":0.054680237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":882,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:56.138549+0000","last_change":"2023-10-30T20:25:36.297471+0000","last_active":"2023-10-30T20:45:56.138549+0000","last_peered":"2023-10-30T20:45:56.138549+0000","last_clean":"2023-10-30T20:45:56.138549+0000","last_became_active":"2023-10-30T20:25:36.286902+0000","last_became_peered":"2023-10-30T20:25:36.286902+0000","last_unstale":"2023-10-30T20:45:56.138549+0000","last_undegraded":"2023-10-30T20:45:56.138549+0000","last_fullsized":"2023-10-30T20:45:56.138549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":867,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:03.143318+0000","last_change":"2023-10-30T20:46:03.144344+0000","last_active":"2023-10-30T20:46:03.143318+0000","last_peered":"2023-10-30T20:46:03.143318+0000","last_clean":"2023-10-30T20:46:03.143318+0000","last_became_active":"2023-10-30T20:25:25.367753+0000","last_became_peered":"2023-10-30T20:25:25.367753+0000","last_unstale":"2023-10-30T20:46:03.143318+0000","last_undegraded":"2023-10-30T20:46:03.143318+0000","last_fullsized":"2023-10-30T20:46:03.143318+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:03.141961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:03.141961+0000","last_clean_scrub_stamp":"2023-10-30T20:46:03.141961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:54:33.638458+0000","scrub_duration":0.062460834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":866,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:06.364710+0000","last_change":"2023-10-30T20:46:06.364710+0000","last_active":"2023-10-30T20:46:06.364710+0000","last_peered":"2023-10-30T20:46:06.364710+0000","last_clean":"2023-10-30T20:46:06.364710+0000","last_became_active":"2023-10-30T20:25:33.362871+0000","last_became_peered":"2023-10-30T20:25:33.362871+0000","last_unstale":"2023-10-30T20:46:06.364710+0000","last_undegraded":"2023-10-30T20:46:06.364710+0000","last_fullsized":"2023-10-30T20:46:06.364710+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:06.362349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:06.362349+0000","last_clean_scrub_stamp":"2023-10-30T20:46:06.362349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:18.646609+0000","scrub_duration":0.064646558000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":866,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:07.344656+0000","last_change":"2023-10-30T20:46:07.345677+0000","last_active":"2023-10-30T20:46:07.344656+0000","last_peered":"2023-10-30T20:46:07.344656+0000","last_clean":"2023-10-30T20:46:07.344656+0000","last_became_active":"2023-10-30T20:25:35.047116+0000","last_became_peered":"2023-10-30T20:25:35.047116+0000","last_unstale":"2023-10-30T20:46:07.344656+0000","last_undegraded":"2023-10-30T20:46:07.344656+0000","last_fullsized":"2023-10-30T20:46:07.344656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:07.343285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:07.343285+0000","last_clean_scrub_stamp":"2023-10-30T20:46:07.343285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:49:32.952530+0000","scrub_duration":0.038919666999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":872,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.711102+0000","last_change":"2023-10-30T20:25:45.287217+0000","last_active":"2023-10-30T20:45:55.711102+0000","last_peered":"2023-10-30T20:45:55.711102+0000","last_clean":"2023-10-30T20:45:55.711102+0000","last_became_active":"2023-10-30T20:25:45.271565+0000","last_became_peered":"2023-10-30T20:25:45.271565+0000","last_unstale":"2023-10-30T20:45:55.711102+0000","last_undegraded":"2023-10-30T20:45:55.711102+0000","last_fullsized":"2023-10-30T20:45:55.711102+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":873,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:55.722505+0000","last_change":"2023-10-30T20:25:28.837575+0000","last_active":"2023-10-30T20:45:55.722505+0000","last_peered":"2023-10-30T20:45:55.722505+0000","last_clean":"2023-10-30T20:45:55.722505+0000","last_became_active":"2023-10-30T20:25:28.829991+0000","last_became_peered":"2023-10-30T20:25:28.829991+0000","last_unstale":"2023-10-30T20:45:55.722505+0000","last_undegraded":"2023-10-30T20:45:55.722505+0000","last_fullsized":"2023-10-30T20:45:55.722505+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:25:17.587663+0000","last_clean_scrub_stamp":"2023-10-30T20:25:17.587663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":886,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:59.550815+0000","last_change":"2023-10-30T20:45:59.551856+0000","last_active":"2023-10-30T20:45:59.550815+0000","last_peered":"2023-10-30T20:45:59.550815+0000","last_clean":"2023-10-30T20:45:59.550815+0000","last_became_active":"2023-10-30T20:24:04.692769+0000","last_became_peered":"2023-10-30T20:24:04.692769+0000","last_unstale":"2023-10-30T20:45:59.550815+0000","last_undegraded":"2023-10-30T20:45:59.550815+0000","last_fullsized":"2023-10-30T20:45:59.550815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:59.549436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:59.549436+0000","last_clean_scrub_stamp":"2023-10-30T20:45:59.549436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T06:18:24.134116+0000","scrub_duration":0.076865441000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":857,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:58.476474+0000","last_change":"2023-10-30T20:45:58.477487+0000","last_active":"2023-10-30T20:45:58.476474+0000","last_peered":"2023-10-30T20:45:58.476474+0000","last_clean":"2023-10-30T20:45:58.476474+0000","last_became_active":"2023-10-30T20:25:27.291944+0000","last_became_peered":"2023-10-30T20:25:27.291944+0000","last_unstale":"2023-10-30T20:45:58.476474+0000","last_undegraded":"2023-10-30T20:45:58.476474+0000","last_fullsized":"2023-10-30T20:45:58.476474+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:58.475059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:58.475059+0000","last_clean_scrub_stamp":"2023-10-30T20:45:58.475059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:37:41.642031+0000","scrub_duration":0.045242620999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":884,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:59.837045+0000","last_change":"2023-10-30T20:45:59.838090+0000","last_active":"2023-10-30T20:45:59.837045+0000","last_peered":"2023-10-30T20:45:59.837045+0000","last_clean":"2023-10-30T20:45:59.837045+0000","last_became_active":"2023-10-30T20:24:04.712006+0000","last_became_peered":"2023-10-30T20:24:04.712006+0000","last_unstale":"2023-10-30T20:45:59.837045+0000","last_undegraded":"2023-10-30T20:45:59.837045+0000","last_fullsized":"2023-10-30T20:45:59.837045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:59.835644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:59.835644+0000","last_clean_scrub_stamp":"2023-10-30T20:45:59.835644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:21:20.761019+0000","scrub_duration":0.035908962000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":858,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:05.604073+0000","last_change":"2023-10-30T20:46:05.605032+0000","last_active":"2023-10-30T20:46:05.604073+0000","last_peered":"2023-10-30T20:46:05.604073+0000","last_clean":"2023-10-30T20:46:05.604073+0000","last_became_active":"2023-10-30T20:25:39.852846+0000","last_became_peered":"2023-10-30T20:25:39.852846+0000","last_unstale":"2023-10-30T20:46:05.604073+0000","last_undegraded":"2023-10-30T20:46:05.604073+0000","last_fullsized":"2023-10-30T20:46:05.604073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:05.602716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:05.602716+0000","last_clean_scrub_stamp":"2023-10-30T20:46:05.602716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:09:03.196298+0000","scrub_duration":0.048731882999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":886,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:07.540990+0000","last_change":"2023-10-30T20:46:07.542035+0000","last_active":"2023-10-30T20:46:07.540990+0000","last_peered":"2023-10-30T20:46:07.540990+0000","last_clean":"2023-10-30T20:46:07.540990+0000","last_became_active":"2023-10-30T20:24:04.310065+0000","last_became_peered":"2023-10-30T20:24:04.310065+0000","last_unstale":"2023-10-30T20:46:07.540990+0000","last_undegraded":"2023-10-30T20:46:07.540990+0000","last_fullsized":"2023-10-30T20:46:07.540990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:07.539599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:07.539599+0000","last_clean_scrub_stamp":"2023-10-30T20:46:07.539599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:21:19.638528+0000","scrub_duration":0.034526423000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":867,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:01.356922+0000","last_change":"2023-10-30T20:46:01.357982+0000","last_active":"2023-10-30T20:46:01.356922+0000","last_peered":"2023-10-30T20:46:01.356922+0000","last_clean":"2023-10-30T20:46:01.356922+0000","last_became_active":"2023-10-30T20:25:42.322505+0000","last_became_peered":"2023-10-30T20:25:42.322505+0000","last_unstale":"2023-10-30T20:46:01.356922+0000","last_undegraded":"2023-10-30T20:46:01.356922+0000","last_fullsized":"2023-10-30T20:46:01.356922+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:01.355555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:01.355555+0000","last_clean_scrub_stamp":"2023-10-30T20:46:01.355555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:24:13.843581+0000","scrub_duration":0.035547902999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":871,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:00.012998+0000","last_change":"2023-10-30T20:46:00.014043+0000","last_active":"2023-10-30T20:46:00.012998+0000","last_peered":"2023-10-30T20:46:00.012998+0000","last_clean":"2023-10-30T20:46:00.012998+0000","last_became_active":"2023-10-30T20:24:04.525673+0000","last_became_peered":"2023-10-30T20:24:04.525673+0000","last_unstale":"2023-10-30T20:46:00.012998+0000","last_undegraded":"2023-10-30T20:46:00.012998+0000","last_fullsized":"2023-10-30T20:46:00.012998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:00.011531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:00.011531+0000","last_clean_scrub_stamp":"2023-10-30T20:46:00.011531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:55:31.626108+0000","scrub_duration":0.034773421999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":867,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:00.336217+0000","last_change":"2023-10-30T20:46:00.338214+0000","last_active":"2023-10-30T20:46:00.336217+0000","last_peered":"2023-10-30T20:46:00.336217+0000","last_clean":"2023-10-30T20:46:00.336217+0000","last_became_active":"2023-10-30T20:25:33.242618+0000","last_became_peered":"2023-10-30T20:25:33.242618+0000","last_unstale":"2023-10-30T20:46:00.336217+0000","last_undegraded":"2023-10-30T20:46:00.336217+0000","last_fullsized":"2023-10-30T20:46:00.336217+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:00.329774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:00.329774+0000","last_clean_scrub_stamp":"2023-10-30T20:46:00.329774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:30:04.958207+0000","scrub_duration":0.033263122999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":886,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:02.508415+0000","last_change":"2023-10-30T20:46:02.510511+0000","last_active":"2023-10-30T20:46:02.508415+0000","last_peered":"2023-10-30T20:46:02.508415+0000","last_clean":"2023-10-30T20:46:02.508415+0000","last_became_active":"2023-10-30T20:24:04.512341+0000","last_became_peered":"2023-10-30T20:24:04.512341+0000","last_unstale":"2023-10-30T20:46:02.508415+0000","last_undegraded":"2023-10-30T20:46:02.508415+0000","last_fullsized":"2023-10-30T20:46:02.508415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:02.504542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:02.504542+0000","last_clean_scrub_stamp":"2023-10-30T20:46:02.504542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:17.334919+0000","scrub_duration":0.031536596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":858,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:59.517184+0000","last_change":"2023-10-30T20:45:59.521395+0000","last_active":"2023-10-30T20:45:59.517184+0000","last_peered":"2023-10-30T20:45:59.517184+0000","last_clean":"2023-10-30T20:45:59.517184+0000","last_became_active":"2023-10-30T20:25:25.151444+0000","last_became_peered":"2023-10-30T20:25:25.151444+0000","last_unstale":"2023-10-30T20:45:59.517184+0000","last_undegraded":"2023-10-30T20:45:59.517184+0000","last_fullsized":"2023-10-30T20:45:59.517184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:59.515798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:59.515798+0000","last_clean_scrub_stamp":"2023-10-30T20:45:59.515798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:50:26.591499+0000","scrub_duration":0.057961194000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":883,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:04.891262+0000","last_change":"2023-10-30T20:46:04.897625+0000","last_active":"2023-10-30T20:46:04.891262+0000","last_peered":"2023-10-30T20:46:04.891262+0000","last_clean":"2023-10-30T20:46:04.891262+0000","last_became_active":"2023-10-30T20:24:05.475394+0000","last_became_peered":"2023-10-30T20:24:05.475394+0000","last_unstale":"2023-10-30T20:46:04.891262+0000","last_undegraded":"2023-10-30T20:46:04.891262+0000","last_fullsized":"2023-10-30T20:46:04.891262+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:04.887537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:04.887537+0000","last_clean_scrub_stamp":"2023-10-30T20:46:04.887537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:53:22.116720+0000","scrub_duration":0.195897509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":866,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:01.045323+0000","last_change":"2023-10-30T20:46:01.046349+0000","last_active":"2023-10-30T20:46:01.045323+0000","last_peered":"2023-10-30T20:46:01.045323+0000","last_clean":"2023-10-30T20:46:01.045323+0000","last_became_active":"2023-10-30T20:25:23.473157+0000","last_became_peered":"2023-10-30T20:25:23.473157+0000","last_unstale":"2023-10-30T20:46:01.045323+0000","last_undegraded":"2023-10-30T20:46:01.045323+0000","last_fullsized":"2023-10-30T20:46:01.045323+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:01.043950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:01.043950+0000","last_clean_scrub_stamp":"2023-10-30T20:46:01.043950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:51:31.998471+0000","scrub_duration":0.031882675999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":887,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:04.526972+0000","last_change":"2023-10-30T20:46:04.526972+0000","last_active":"2023-10-30T20:46:04.526972+0000","last_peered":"2023-10-30T20:46:04.526972+0000","last_clean":"2023-10-30T20:46:04.526972+0000","last_became_active":"2023-10-30T20:24:04.254051+0000","last_became_peered":"2023-10-30T20:24:04.254051+0000","last_unstale":"2023-10-30T20:46:04.526972+0000","last_undegraded":"2023-10-30T20:46:04.526972+0000","last_fullsized":"2023-10-30T20:46:04.526972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:04.524295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:04.524295+0000","last_clean_scrub_stamp":"2023-10-30T20:46:04.524295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T08:09:25.634732+0000","scrub_duration":0.029218244000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":890,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:02.733132+0000","last_change":"2023-10-30T20:46:02.736934+0000","last_active":"2023-10-30T20:46:02.733132+0000","last_peered":"2023-10-30T20:46:02.733132+0000","last_clean":"2023-10-30T20:46:02.733132+0000","last_became_active":"2023-10-30T20:25:36.260481+0000","last_became_peered":"2023-10-30T20:25:36.260481+0000","last_unstale":"2023-10-30T20:46:02.733132+0000","last_undegraded":"2023-10-30T20:46:02.733132+0000","last_fullsized":"2023-10-30T20:46:02.733132+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:02.731814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:02.731814+0000","last_clean_scrub_stamp":"2023-10-30T20:46:02.731814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:03:02.181324+0000","scrub_duration":0.056385602999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":886,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:58.542694+0000","last_change":"2023-10-30T20:45:58.543769+0000","last_active":"2023-10-30T20:45:58.542694+0000","last_peered":"2023-10-30T20:45:58.542694+0000","last_clean":"2023-10-30T20:45:58.542694+0000","last_became_active":"2023-10-30T20:24:04.671790+0000","last_became_peered":"2023-10-30T20:24:04.671790+0000","last_unstale":"2023-10-30T20:45:58.542694+0000","last_undegraded":"2023-10-30T20:45:58.542694+0000","last_fullsized":"2023-10-30T20:45:58.542694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:58.541256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:58.541256+0000","last_clean_scrub_stamp":"2023-10-30T20:45:58.541256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:47:21.915073+0000","scrub_duration":0.034256602999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"904'78","reported_seq":1004,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:59.822887+0000","last_change":"2023-10-30T20:45:59.822887+0000","last_active":"2023-10-30T20:45:59.822887+0000","last_peered":"2023-10-30T20:45:59.822887+0000","last_clean":"2023-10-30T20:45:59.822887+0000","last_became_active":"2023-10-30T20:23:57.098872+0000","last_became_peered":"2023-10-30T20:23:57.098872+0000","last_unstale":"2023-10-30T20:45:59.822887+0000","last_undegraded":"2023-10-30T20:45:59.822887+0000","last_fullsized":"2023-10-30T20:45:59.822887+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'78","last_scrub_stamp":"2023-10-30T20:45:59.817552+0000","last_deep_scrub":"904'78","last_deep_scrub_stamp":"2023-10-30T20:45:59.817552+0000","last_clean_scrub_stamp":"2023-10-30T20:45:59.817552+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:18:28.510123+0000","scrub_duration":0.095846543000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"904'8","reported_seq":873,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:02.565655+0000","last_change":"2023-10-30T20:46:02.565655+0000","last_active":"2023-10-30T20:46:02.565655+0000","last_peered":"2023-10-30T20:46:02.565655+0000","last_clean":"2023-10-30T20:46:02.565655+0000","last_became_active":"2023-10-30T20:25:25.362636+0000","last_became_peered":"2023-10-30T20:25:25.362636+0000","last_unstale":"2023-10-30T20:46:02.565655+0000","last_undegraded":"2023-10-30T20:46:02.565655+0000","last_fullsized":"2023-10-30T20:46:02.565655+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'8","last_scrub_stamp":"2023-10-30T20:46:02.560805+0000","last_deep_scrub":"904'8","last_deep_scrub_stamp":"2023-10-30T20:46:02.560805+0000","last_clean_scrub_stamp":"2023-10-30T20:46:02.560805+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:39.733451+0000","scrub_duration":0.073184763999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":868,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:59.300374+0000","last_change":"2023-10-30T20:45:59.302082+0000","last_active":"2023-10-30T20:45:59.300374+0000","last_peered":"2023-10-30T20:45:59.300374+0000","last_clean":"2023-10-30T20:45:59.300374+0000","last_became_active":"2023-10-30T20:25:23.607599+0000","last_became_peered":"2023-10-30T20:25:23.607599+0000","last_unstale":"2023-10-30T20:45:59.300374+0000","last_undegraded":"2023-10-30T20:45:59.300374+0000","last_fullsized":"2023-10-30T20:45:59.300374+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:59.298300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:59.298300+0000","last_clean_scrub_stamp":"2023-10-30T20:45:59.298300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:10:33.842120+0000","scrub_duration":0.034934029999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":880,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:04.855387+0000","last_change":"2023-10-30T20:46:04.855387+0000","last_active":"2023-10-30T20:46:04.855387+0000","last_peered":"2023-10-30T20:46:04.855387+0000","last_clean":"2023-10-30T20:46:04.855387+0000","last_became_active":"2023-10-30T20:25:28.326296+0000","last_became_peered":"2023-10-30T20:25:28.326296+0000","last_unstale":"2023-10-30T20:46:04.855387+0000","last_undegraded":"2023-10-30T20:46:04.855387+0000","last_fullsized":"2023-10-30T20:46:04.855387+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:04.853002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:04.853002+0000","last_clean_scrub_stamp":"2023-10-30T20:46:04.853002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:52:09.667086+0000","scrub_duration":0.042164409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":867,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:04.130324+0000","last_change":"2023-10-30T20:46:04.130324+0000","last_active":"2023-10-30T20:46:04.130324+0000","last_peered":"2023-10-30T20:46:04.130324+0000","last_clean":"2023-10-30T20:46:04.130324+0000","last_became_active":"2023-10-30T20:25:20.671965+0000","last_became_peered":"2023-10-30T20:25:20.671965+0000","last_unstale":"2023-10-30T20:46:04.130324+0000","last_undegraded":"2023-10-30T20:46:04.130324+0000","last_fullsized":"2023-10-30T20:46:04.130324+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:04.124872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:04.124872+0000","last_clean_scrub_stamp":"2023-10-30T20:46:04.124872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T06:23:46.948945+0000","scrub_duration":0.038255145999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":880,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:13.880673+0000","last_change":"2023-10-30T20:46:13.880673+0000","last_active":"2023-10-30T20:46:13.880673+0000","last_peered":"2023-10-30T20:46:13.880673+0000","last_clean":"2023-10-30T20:46:13.880673+0000","last_became_active":"2023-10-30T20:25:28.979668+0000","last_became_peered":"2023-10-30T20:25:28.979668+0000","last_unstale":"2023-10-30T20:46:13.880673+0000","last_undegraded":"2023-10-30T20:46:13.880673+0000","last_fullsized":"2023-10-30T20:46:13.880673+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:13.878449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:13.878449+0000","last_clean_scrub_stamp":"2023-10-30T20:46:13.878449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:11:18.690935+0000","scrub_duration":0.038533900000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":880,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:08.954913+0000","last_change":"2023-10-30T20:46:08.954913+0000","last_active":"2023-10-30T20:46:08.954913+0000","last_peered":"2023-10-30T20:46:08.954913+0000","last_clean":"2023-10-30T20:46:08.954913+0000","last_became_active":"2023-10-30T20:25:23.058176+0000","last_became_peered":"2023-10-30T20:25:23.058176+0000","last_unstale":"2023-10-30T20:46:08.954913+0000","last_undegraded":"2023-10-30T20:46:08.954913+0000","last_fullsized":"2023-10-30T20:46:08.954913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:08.949899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:08.949899+0000","last_clean_scrub_stamp":"2023-10-30T20:46:08.949899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T06:23:50.453541+0000","scrub_duration":0.028102695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":881,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:09.652925+0000","last_change":"2023-10-30T20:46:09.653991+0000","last_active":"2023-10-30T20:46:09.652925+0000","last_peered":"2023-10-30T20:46:09.652925+0000","last_clean":"2023-10-30T20:46:09.652925+0000","last_became_active":"2023-10-30T20:25:28.798013+0000","last_became_peered":"2023-10-30T20:25:28.798013+0000","last_unstale":"2023-10-30T20:46:09.652925+0000","last_undegraded":"2023-10-30T20:46:09.652925+0000","last_fullsized":"2023-10-30T20:46:09.652925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:09.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:09.651479+0000","last_clean_scrub_stamp":"2023-10-30T20:46:09.651479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:34:16.105390+0000","scrub_duration":0.052863632000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":881,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:10.624227+0000","last_change":"2023-10-30T20:46:10.625301+0000","last_active":"2023-10-30T20:46:10.624227+0000","last_peered":"2023-10-30T20:46:10.624227+0000","last_clean":"2023-10-30T20:46:10.624227+0000","last_became_active":"2023-10-30T20:25:23.709112+0000","last_became_peered":"2023-10-30T20:25:23.709112+0000","last_unstale":"2023-10-30T20:46:10.624227+0000","last_undegraded":"2023-10-30T20:46:10.624227+0000","last_fullsized":"2023-10-30T20:46:10.624227+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:10.622841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:10.622841+0000","last_clean_scrub_stamp":"2023-10-30T20:46:10.622841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:21:16.161770+0000","scrub_duration":0.037042064999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":861,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:05.279023+0000","last_change":"2023-10-30T20:46:05.280404+0000","last_active":"2023-10-30T20:46:05.279023+0000","last_peered":"2023-10-30T20:46:05.279023+0000","last_clean":"2023-10-30T20:46:05.279023+0000","last_became_active":"2023-10-30T20:25:28.762953+0000","last_became_peered":"2023-10-30T20:25:28.762953+0000","last_unstale":"2023-10-30T20:46:05.279023+0000","last_undegraded":"2023-10-30T20:46:05.279023+0000","last_fullsized":"2023-10-30T20:46:05.279023+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:05.277120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:05.277120+0000","last_clean_scrub_stamp":"2023-10-30T20:46:05.277120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:09:47.711405+0000","scrub_duration":0.050602715999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":878,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:00.883410+0000","last_change":"2023-10-30T20:46:00.884420+0000","last_active":"2023-10-30T20:46:00.883410+0000","last_peered":"2023-10-30T20:46:00.883410+0000","last_clean":"2023-10-30T20:46:00.883410+0000","last_became_active":"2023-10-30T20:25:21.911569+0000","last_became_peered":"2023-10-30T20:25:21.911569+0000","last_unstale":"2023-10-30T20:46:00.883410+0000","last_undegraded":"2023-10-30T20:46:00.883410+0000","last_fullsized":"2023-10-30T20:46:00.883410+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:00.882062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:00.882062+0000","last_clean_scrub_stamp":"2023-10-30T20:46:00.882062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:34:26.631811+0000","scrub_duration":0.032802308000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":885,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:06.934699+0000","last_change":"2023-10-30T20:46:06.935680+0000","last_active":"2023-10-30T20:46:06.934699+0000","last_peered":"2023-10-30T20:46:06.934699+0000","last_clean":"2023-10-30T20:46:06.934699+0000","last_became_active":"2023-10-30T20:29:14.860294+0000","last_became_peered":"2023-10-30T20:29:14.860294+0000","last_unstale":"2023-10-30T20:46:06.934699+0000","last_undegraded":"2023-10-30T20:46:06.934699+0000","last_fullsized":"2023-10-30T20:46:06.934699+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:06.933427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:06.933427+0000","last_clean_scrub_stamp":"2023-10-30T20:46:06.933427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:03:24.607098+0000","scrub_duration":0.059309705999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":889,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:04.771463+0000","last_change":"2023-10-30T20:46:04.772433+0000","last_active":"2023-10-30T20:46:04.771463+0000","last_peered":"2023-10-30T20:46:04.771463+0000","last_clean":"2023-10-30T20:46:04.771463+0000","last_became_active":"2023-10-30T20:25:29.004167+0000","last_became_peered":"2023-10-30T20:25:29.004167+0000","last_unstale":"2023-10-30T20:46:04.771463+0000","last_undegraded":"2023-10-30T20:46:04.771463+0000","last_fullsized":"2023-10-30T20:46:04.771463+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:04.770139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:04.770139+0000","last_clean_scrub_stamp":"2023-10-30T20:46:04.770139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:54:39.727192+0000","scrub_duration":0.035219334999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477006,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59864,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311976,"statfs":{"total":96636764160,"available":96575463424,"internally_reserved":0,"allocated":6635520,"data_stored":3147947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":25.940000000000001,"5min":18.199999999999999,"15min":16.863},"min":{"1min":5.1040000000000001,"5min":5.0529999999999999,"15min":3.9740000000000002},"max":{"1min":156.071,"5min":222.095,"15min":244.999},"last":5.7839999999999998},{"interface":"front","average":{"1min":32.825000000000003,"5min":20.239000000000001,"15min":18.428000000000001},"min":{"1min":5.1020000000000003,"5min":4.9279999999999999,"15min":4.2539999999999996},"max":{"1min":219.797,"5min":219.797,"15min":219.797},"last":10.972}]},{"osd":1,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":25.375,"5min":20.100999999999999,"15min":20.373000000000001},"min":{"1min":9.6869999999999994,"5min":5.6500000000000004,"15min":4.5940000000000003},"max":{"1min":143.87200000000001,"5min":231.85499999999999,"15min":256.815},"last":14.18},{"interface":"front","average":{"1min":22.506,"5min":19.602,"15min":22.550999999999998},"min":{"1min":7.298,"5min":5.7320000000000002,"15min":5.726},"max":{"1min":122.70399999999999,"5min":237.88300000000001,"15min":424.57999999999998},"last":8.7240000000000002}]},{"osd":2,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":34.981999999999999,"5min":24.867999999999999,"15min":24.663},"min":{"1min":9.7789999999999999,"5min":6.1539999999999999,"15min":6.1539999999999999},"max":{"1min":267.81099999999998,"5min":267.81099999999998,"15min":835.96799999999996},"last":16.774000000000001},{"interface":"front","average":{"1min":37.021999999999998,"5min":26.030999999999999,"15min":23.010000000000002},"min":{"1min":10.771000000000001,"5min":10.173,"15min":8.2479999999999993},"max":{"1min":244.50399999999999,"5min":247.178,"15min":261.50900000000001},"last":17.559000000000001}]},{"osd":4,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":29.966000000000001,"5min":27.501000000000001,"15min":23.704999999999998},"min":{"1min":6.8390000000000004,"5min":6.625,"15min":5.5190000000000001},"max":{"1min":196.904,"5min":247.904,"15min":247.904},"last":14.946999999999999},{"interface":"front","average":{"1min":29.170999999999999,"5min":25.465,"15min":24.071000000000002},"min":{"1min":7.9900000000000002,"5min":6.8620000000000001,"15min":6.3840000000000003},"max":{"1min":210.88200000000001,"5min":235.64099999999999,"15min":235.64099999999999},"last":9.6679999999999993}]},{"osd":5,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":35.137999999999998,"5min":26.385999999999999,"15min":25.082999999999998},"min":{"1min":8.4100000000000001,"5min":6.726,"15min":6.726},"max":{"1min":198.76300000000001,"5min":233.28999999999999,"15min":233.28999999999999},"last":12.772},{"interface":"front","average":{"1min":46.277000000000001,"5min":31.02,"15min":24.657},"min":{"1min":11.351000000000001,"5min":10.645,"15min":7.883},"max":{"1min":227.28299999999999,"5min":244.23500000000001,"15min":244.23500000000001},"last":18.454999999999998}]},{"osd":6,"last update":"Mon Oct 30 20:45:27 2023","interfaces":[{"interface":"back","average":{"1min":41.406999999999996,"5min":30.888000000000002,"15min":27.672999999999998},"min":{"1min":11.547000000000001,"5min":9.1989999999999998,"15min":9.1989999999999998},"max":{"1min":269.15600000000001,"5min":269.15600000000001,"15min":269.15600000000001},"last":19.228000000000002},{"interface":"front","average":{"1min":32.853999999999999,"5min":24.702999999999999,"15min":22.295999999999999},"min":{"1min":6.093,"5min":6.093,"15min":6.093},"max":{"1min":262.53500000000003,"5min":262.53500000000003,"15min":262.53500000000003},"last":13.688000000000001}]},{"osd":7,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":29.765999999999998,"5min":27.036000000000001,"15min":24.699000000000002},"min":{"1min":9.7669999999999995,"5min":8.5570000000000004,"15min":7.3799999999999999},"max":{"1min":222.32400000000001,"5min":264.48000000000002,"15min":264.48000000000002},"last":15.754},{"interface":"front","average":{"1min":31.981999999999999,"5min":27.591000000000001,"15min":26.734999999999999},"min":{"1min":9.2919999999999998,"5min":8.9700000000000006,"15min":8.9700000000000006},"max":{"1min":233.99700000000001,"5min":245.042,"15min":245.042},"last":10.364000000000001}]}]},{"osd":1,"up_from":16,"seq":68719477007,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64060,"kb_used_data":6516,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94307780,"statfs":{"total":96636764160,"available":96571166720,"internally_reserved":0,"allocated":6672384,"data_stored":3545787,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.762,"5min":15.228999999999999,"15min":16.553999999999998},"min":{"1min":4.1130000000000004,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":98.356999999999999,"5min":244.65000000000001,"15min":244.65000000000001},"last":6.0499999999999998},{"interface":"front","average":{"1min":11.536,"5min":12.617000000000001,"15min":18.32},"min":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":3.847},"max":{"1min":39.68,"5min":65.853999999999999,"15min":407.38900000000001},"last":9.1240000000000006}]},{"osd":2,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":18.402000000000001,"5min":16.823,"15min":19.931000000000001},"min":{"1min":7.5389999999999997,"5min":7.367,"15min":4.1500000000000004},"max":{"1min":80.688000000000002,"5min":113.649,"15min":297.26499999999999},"last":10.429},{"interface":"front","average":{"1min":23.346,"5min":19.442,"15min":20.483000000000001},"min":{"1min":9.8040000000000003,"5min":8.5690000000000008,"15min":6.5469999999999997},"max":{"1min":148.03999999999999,"5min":148.03999999999999,"15min":167.12100000000001},"last":14.925000000000001}]},{"osd":3,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":17.946000000000002,"5min":19.791,"15min":21.096},"min":{"1min":8.8569999999999993,"5min":7.8029999999999999,"15min":6.5990000000000002},"max":{"1min":84.090999999999994,"5min":150.166,"15min":213.971},"last":11.010999999999999},{"interface":"front","average":{"1min":19.449000000000002,"5min":22.609000000000002,"15min":24.210999999999999},"min":{"1min":12.093999999999999,"5min":9.8409999999999993,"15min":8.1449999999999996},"max":{"1min":74.275999999999996,"5min":146.15899999999999,"15min":223.49199999999999},"last":15.833}]},{"osd":4,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":13.090999999999999,"5min":20.369,"15min":21.076000000000001},"min":{"1min":8.1349999999999998,"5min":6.8659999999999997,"15min":5.9809999999999999},"max":{"1min":30.533000000000001,"5min":200.98400000000001,"15min":215.435},"last":7.1769999999999996},{"interface":"front","average":{"1min":17.289000000000001,"5min":23.798999999999999,"15min":25.364000000000001},"min":{"1min":12.874000000000001,"5min":11.516999999999999,"15min":8.9939999999999998},"max":{"1min":26.600000000000001,"5min":192.69200000000001,"15min":197.34},"last":16.614999999999998}]},{"osd":5,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":13.349,"5min":19.747,"15min":19.356000000000002},"min":{"1min":7.4020000000000001,"5min":6.4580000000000002,"15min":5.1390000000000002},"max":{"1min":62.798000000000002,"5min":209.35499999999999,"15min":241.798},"last":7.7850000000000001},{"interface":"front","average":{"1min":17.783000000000001,"5min":20.292000000000002,"15min":21.498000000000001},"min":{"1min":9.5739999999999998,"5min":6.6230000000000002,"15min":6.6230000000000002},"max":{"1min":68.882000000000005,"5min":118.65600000000001,"15min":248.124},"last":11.603999999999999}]},{"osd":6,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.948,"5min":20.902000000000001,"15min":23.236999999999998},"min":{"1min":10.205,"5min":7.7220000000000004,"15min":6.9880000000000004},"max":{"1min":37.037999999999997,"5min":204.71899999999999,"15min":406.72399999999999},"last":9.8350000000000009},{"interface":"front","average":{"1min":28.690999999999999,"5min":25.748000000000001,"15min":25.416},"min":{"1min":11.704000000000001,"5min":10.397,"15min":8.4749999999999996},"max":{"1min":179.98099999999999,"5min":179.98099999999999,"15min":179.98099999999999},"last":17.356000000000002}]},{"osd":7,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":14.643000000000001,"5min":18.478999999999999,"15min":23.765000000000001},"min":{"1min":6.0410000000000004,"5min":6.0410000000000004,"15min":6.0110000000000001},"max":{"1min":58.841999999999999,"5min":188.792,"15min":249.88300000000001},"last":8.3620000000000001},{"interface":"front","average":{"1min":19.361000000000001,"5min":21.233000000000001,"15min":25.628},"min":{"1min":12.742000000000001,"5min":8.8559999999999999,"15min":7.1360000000000001},"max":{"1min":59.718000000000004,"5min":134.89400000000001,"15min":275.36700000000002},"last":12.236000000000001}]}]},{"osd":0,"up_from":16,"seq":68719477007,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59868,"kb_used_data":6484,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311972,"statfs":{"total":96636764160,"available":96575459328,"internally_reserved":0,"allocated":6639616,"data_stored":3147952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":14.162000000000001,"5min":13.208,"15min":18.428000000000001},"min":{"1min":5.508,"5min":4.1740000000000004,"15min":4.0819999999999999},"max":{"1min":64.745999999999995,"5min":145.857,"15min":341.85300000000001},"last":10.228999999999999},{"interface":"front","average":{"1min":15.507,"5min":16.523,"15min":18.965},"min":{"1min":5.9400000000000004,"5min":5.798,"15min":5.1669999999999998},"max":{"1min":71.578000000000003,"5min":196.148,"15min":243.62100000000001},"last":12.91}]},{"osd":2,"last update":"Mon Oct 30 20:46:01 2023","interfaces":[{"interface":"back","average":{"1min":18.695,"5min":19.710999999999999,"15min":25.018000000000001},"min":{"1min":6.0309999999999997,"5min":5.298,"15min":4.6449999999999996},"max":{"1min":93.063000000000002,"5min":199.71100000000001,"15min":580.43100000000004},"last":11.148999999999999},{"interface":"front","average":{"1min":31.413,"5min":21.189,"15min":26.045000000000002},"min":{"1min":6.0910000000000002,"5min":6.0910000000000002,"15min":6.0910000000000002},"max":{"1min":292.03699999999998,"5min":292.03699999999998,"15min":1090.171},"last":14.316000000000001}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.609,"5min":16.076000000000001,"15min":16.977},"min":{"1min":6.2199999999999998,"5min":5.298,"15min":5.298},"max":{"1min":76.861000000000004,"5min":148.31299999999999,"15min":167.607},"last":5.7789999999999999},{"interface":"front","average":{"1min":30.317,"5min":25.641999999999999,"15min":24.917999999999999},"min":{"1min":7.1719999999999997,"5min":7.1719999999999997,"15min":6.4370000000000003},"max":{"1min":290.19600000000003,"5min":290.19600000000003,"15min":318.245},"last":9.5709999999999997}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":24.504000000000001,"5min":20.890000000000001,"15min":20.295000000000002},"min":{"1min":7.234,"5min":7.1059999999999999,"15min":5.8170000000000002},"max":{"1min":116.98399999999999,"5min":174.67699999999999,"15min":178.642},"last":7.8600000000000003},{"interface":"front","average":{"1min":30.776,"5min":28.367999999999999,"15min":27.843},"min":{"1min":8.3770000000000007,"5min":8.1010000000000009,"15min":6.2960000000000003},"max":{"1min":153.124,"5min":219.14699999999999,"15min":242.06800000000001},"last":15.506}]},{"osd":5,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.25,"5min":23.315999999999999,"15min":22.43},"min":{"1min":9.3379999999999992,"5min":8.3480000000000008,"15min":6.8099999999999996},"max":{"1min":68.546999999999997,"5min":170.024,"15min":170.024},"last":13.613},{"interface":"front","average":{"1min":25.843,"5min":23.294,"15min":22.381},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.8280000000000003},"max":{"1min":138.041,"5min":177.41800000000001,"15min":177.41800000000001},"last":8.6329999999999991}]},{"osd":6,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":27.763999999999999,"5min":29.074999999999999,"15min":30.167000000000002},"min":{"1min":10.817,"5min":8.7599999999999998,"15min":8.2590000000000003},"max":{"1min":151.73500000000001,"5min":189.21899999999999,"15min":330.37799999999999},"last":17.635999999999999},{"interface":"front","average":{"1min":19.260000000000002,"5min":20.901,"15min":24.004000000000001},"min":{"1min":6.8410000000000002,"5min":6.8410000000000002,"15min":6.8410000000000002},"max":{"1min":73.170000000000002,"5min":169.447,"15min":201.50299999999999},"last":14.920999999999999}]},{"osd":7,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":20.75,"5min":23.949999999999999,"15min":27.219999999999999},"min":{"1min":11.298,"5min":10.81,"15min":9.6820000000000004},"max":{"1min":70.683000000000007,"5min":175.667,"15min":268.18900000000002},"last":16.109000000000002},{"interface":"front","average":{"1min":21.437999999999999,"5min":21.204999999999998,"15min":25.626000000000001},"min":{"1min":8.8689999999999998,"5min":8.8689999999999998,"15min":7.3559999999999999},"max":{"1min":77.927000000000007,"5min":173.441,"15min":228.495},"last":11.785}]}]},{"osd":6,"up_from":15,"seq":64424509711,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59864,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311976,"statfs":{"total":96636764160,"available":96575463424,"internally_reserved":0,"allocated":6635520,"data_stored":3147947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:49 2023","interfaces":[{"interface":"back","average":{"1min":15.789,"5min":12.747,"15min":13.856999999999999},"min":{"1min":4.3819999999999997,"5min":4.0179999999999998,"15min":3.7080000000000002},"max":{"1min":66.393000000000001,"5min":73.316000000000003,"15min":151.00299999999999},"last":9.1809999999999992},{"interface":"front","average":{"1min":10.926,"5min":11.757,"15min":15.211},"min":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"max":{"1min":53.823999999999998,"5min":75.060000000000002,"15min":146.49299999999999},"last":7.117}]},{"osd":1,"last update":"Mon Oct 30 20:45:49 2023","interfaces":[{"interface":"back","average":{"1min":14.625,"5min":16.163,"15min":19.315999999999999},"min":{"1min":5.8049999999999997,"5min":5.4379999999999997,"15min":4.9199999999999999},"max":{"1min":74.001000000000005,"5min":117.193,"15min":218.78100000000001},"last":7.7960000000000003},{"interface":"front","average":{"1min":10.275,"5min":13.943,"15min":15.449},"min":{"1min":5.7229999999999999,"5min":5.1379999999999999,"15min":5.0149999999999997},"max":{"1min":27.335999999999999,"5min":128.50200000000001,"15min":155.27199999999999},"last":6.4169999999999998}]},{"osd":2,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":19.477,"5min":16.728000000000002,"15min":21.635000000000002},"min":{"1min":7.9080000000000004,"5min":6.1870000000000003,"15min":6.1870000000000003},"max":{"1min":67.552999999999997,"5min":122.307,"15min":222.303},"last":10.550000000000001},{"interface":"front","average":{"1min":16.923999999999999,"5min":18.561,"15min":26.841000000000001},"min":{"1min":7.9580000000000002,"5min":7.9580000000000002,"15min":6.1820000000000004},"max":{"1min":46.755000000000003,"5min":98.781000000000006,"15min":438.19200000000001},"last":9.8670000000000009}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.385000000000002,"5min":15.81,"15min":17.850000000000001},"min":{"1min":6.4859999999999998,"5min":6.2050000000000001,"15min":6.0179999999999998},"max":{"1min":136.16800000000001,"5min":136.16800000000001,"15min":217.99000000000001},"last":8.577},{"interface":"front","average":{"1min":30.402000000000001,"5min":20.652999999999999,"15min":21.940000000000001},"min":{"1min":6.3890000000000002,"5min":6.3890000000000002,"15min":6.0010000000000003},"max":{"1min":270.25599999999997,"5min":270.25599999999997,"15min":270.25599999999997},"last":11.375}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":19.241,"5min":21.978000000000002,"15min":23.396000000000001},"min":{"1min":11.669,"5min":8.1539999999999999,"15min":7.5800000000000001},"max":{"1min":69.150000000000006,"5min":168.38800000000001,"15min":190.29599999999999},"last":14.762},{"interface":"front","average":{"1min":14.085000000000001,"5min":20.285,"15min":22.73},"min":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"max":{"1min":48.207000000000001,"5min":163.41800000000001,"15min":189.59399999999999},"last":15.41}]},{"osd":5,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":21.626999999999999,"5min":20.792999999999999,"15min":22.007999999999999},"min":{"1min":9.1430000000000007,"5min":8.9939999999999998,"15min":8.9930000000000003},"max":{"1min":68.114000000000004,"5min":133.73400000000001,"15min":138.38800000000001},"last":16.751999999999999},{"interface":"front","average":{"1min":25.091000000000001,"5min":24.527000000000001,"15min":25.061},"min":{"1min":12.673999999999999,"5min":11.9,"15min":10.571},"max":{"1min":68.138999999999996,"5min":150.35499999999999,"15min":152.40199999999999},"last":17.469000000000001}]},{"osd":7,"last update":"Mon Oct 30 20:45:33 2023","interfaces":[{"interface":"back","average":{"1min":17.923999999999999,"5min":18.893999999999998,"15min":23.824999999999999},"min":{"1min":8.8140000000000001,"5min":7.1699999999999999,"15min":6.226},"max":{"1min":64.828999999999994,"5min":98.503,"15min":253.16900000000001},"last":13.352},{"interface":"front","average":{"1min":19.021000000000001,"5min":21.308,"15min":20.143000000000001},"min":{"1min":7.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"max":{"1min":75.846000000000004,"5min":174.071,"15min":221.28399999999999},"last":14.087999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509709,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77948,"kb_used_data":6516,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293892,"statfs":{"total":96636764160,"available":96556945408,"internally_reserved":0,"allocated":6672384,"data_stored":3545787,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":17.625,"5min":18.785,"15min":16.260999999999999},"min":{"1min":5.2329999999999997,"5min":3.8359999999999999,"15min":3.8359999999999999},"max":{"1min":85.513999999999996,"5min":220.148,"15min":220.148},"last":8.4179999999999993},{"interface":"front","average":{"1min":27.861000000000001,"5min":23.675000000000001,"15min":20.949000000000002},"min":{"1min":8.6039999999999992,"5min":5.6059999999999999,"15min":5.6059999999999999},"max":{"1min":145.43000000000001,"5min":177.91,"15min":177.91},"last":9.7910000000000004}]},{"osd":1,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":20.715,"5min":16.585000000000001,"15min":20.353999999999999},"min":{"1min":6.3040000000000003,"5min":4.3010000000000002,"15min":4.3010000000000002},"max":{"1min":71.460999999999999,"5min":74.081000000000003,"15min":293.75400000000002},"last":11.148999999999999},{"interface":"front","average":{"1min":20.943999999999999,"5min":16.175000000000001,"15min":18.327999999999999},"min":{"1min":5.46,"5min":4.8949999999999996,"15min":4.8949999999999996},"max":{"1min":80.397000000000006,"5min":80.397000000000006,"15min":210.797},"last":16.527999999999999}]},{"osd":2,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":31.280000000000001,"5min":23.431999999999999,"15min":25.231000000000002},"min":{"1min":6.2480000000000002,"5min":5.0209999999999999,"15min":5.0209999999999999},"max":{"1min":105.449,"5min":143.273,"15min":519.68799999999999},"last":9.0939999999999994},{"interface":"front","average":{"1min":23.294,"5min":20.007000000000001,"15min":22.029},"min":{"1min":5.6479999999999997,"5min":5.6479999999999997,"15min":5.226},"max":{"1min":82.042000000000002,"5min":154.083,"15min":218.59899999999999},"last":7.2800000000000002}]},{"osd":3,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":41.817,"5min":31.161999999999999,"15min":26.263000000000002},"min":{"1min":9.9900000000000002,"5min":9.9900000000000002,"15min":7.6150000000000002},"max":{"1min":269.52999999999997,"5min":269.52999999999997,"15min":269.52999999999997},"last":13.989000000000001},{"interface":"front","average":{"1min":24.488,"5min":22.629000000000001,"15min":22.704000000000001},"min":{"1min":7.4480000000000004,"5min":6.218,"15min":5.9119999999999999},"max":{"1min":178.78700000000001,"5min":178.78700000000001,"15min":249.797},"last":6.9420000000000002}]},{"osd":4,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":21.725999999999999,"5min":26.437999999999999,"15min":25.25},"min":{"1min":8.1080000000000005,"5min":7.1580000000000004,"15min":6.6100000000000003},"max":{"1min":78.555999999999997,"5min":215.97999999999999,"15min":291.92599999999999},"last":10.449},{"interface":"front","average":{"1min":26.097000000000001,"5min":33.600000000000001,"15min":28.585000000000001},"min":{"1min":11.895,"5min":8.0050000000000008,"15min":6.9550000000000001},"max":{"1min":70.290999999999997,"5min":324.31,"15min":324.31},"last":17.768999999999998}]},{"osd":6,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":32.883000000000003,"5min":28.116,"15min":26.379999999999999},"min":{"1min":10.593,"5min":8.6489999999999991,"15min":7.2199999999999998},"max":{"1min":162.964,"5min":211.255,"15min":251.37700000000001},"last":15.337},{"interface":"front","average":{"1min":38.536000000000001,"5min":33.917999999999999,"15min":28.524000000000001},"min":{"1min":9.3640000000000008,"5min":9.2349999999999994,"15min":7.1619999999999999},"max":{"1min":160.42599999999999,"5min":333.29700000000003,"15min":333.29700000000003},"last":13.353999999999999}]},{"osd":7,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":36.252000000000002,"5min":33.210000000000001,"15min":27.071999999999999},"min":{"1min":11.131,"5min":10.68,"15min":6.1769999999999996},"max":{"1min":116.877,"5min":258.81,"15min":264.52999999999997},"last":11.801},{"interface":"front","average":{"1min":26.748000000000001,"5min":22.239000000000001,"15min":25.803999999999998},"min":{"1min":7.4580000000000002,"5min":5.8410000000000002,"15min":5.8410000000000002},"max":{"1min":90.570999999999998,"5min":154.86799999999999,"15min":255.625},"last":14.699999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542414,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59864,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311976,"statfs":{"total":96636764160,"available":96575463424,"internally_reserved":0,"allocated":6635520,"data_stored":3147947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":9.2010000000000005,"5min":13.369999999999999,"15min":14.285},"min":{"1min":4.9580000000000002,"5min":4.0499999999999998,"15min":3.5920000000000001},"max":{"1min":18.266999999999999,"5min":106.79300000000001,"15min":187.85499999999999},"last":7.9740000000000002},{"interface":"front","average":{"1min":8.7729999999999997,"5min":12.766999999999999,"15min":13.725},"min":{"1min":5.0279999999999996,"5min":4.1020000000000003,"15min":4.1020000000000003},"max":{"1min":15.933,"5min":117.645,"15min":124.342},"last":9.6790000000000003}]},{"osd":1,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":13.706,"5min":18.475999999999999,"15min":19.167999999999999},"min":{"1min":8.1969999999999992,"5min":5.8979999999999997,"15min":5.7409999999999997},"max":{"1min":33.970999999999997,"5min":103.907,"15min":226.63399999999999},"last":11.609999999999999},{"interface":"front","average":{"1min":11.929,"5min":17.565999999999999,"15min":19.605},"min":{"1min":6.5439999999999996,"5min":6.351,"15min":5.859},"max":{"1min":42.719000000000001,"5min":135.41200000000001,"15min":187.24100000000001},"last":10.282}]},{"osd":2,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":17.675000000000001,"5min":18.248999999999999,"15min":20.965},"min":{"1min":7.2869999999999999,"5min":6.2549999999999999,"15min":6.2549999999999999},"max":{"1min":43.064999999999998,"5min":79.168000000000006,"15min":200.285},"last":15.382},{"interface":"front","average":{"1min":26.209,"5min":21.07,"15min":21.149999999999999},"min":{"1min":6.0679999999999996,"5min":5.5090000000000003,"15min":5.5090000000000003},"max":{"1min":211.315,"5min":211.315,"15min":241.15700000000001},"last":17.248999999999999}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":25.942,"5min":21.164000000000001,"15min":22.437999999999999},"min":{"1min":10.516,"5min":8.1509999999999998,"15min":6.6849999999999996},"max":{"1min":205.821,"5min":205.821,"15min":262.43900000000002},"last":14.651},{"interface":"front","average":{"1min":26.780000000000001,"5min":22.042999999999999,"15min":23.443999999999999},"min":{"1min":7.5650000000000004,"5min":6.7460000000000004,"15min":5.9950000000000001},"max":{"1min":225.29900000000001,"5min":225.29900000000001,"15min":330.15300000000002},"last":11.002000000000001}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.153,"5min":21.227,"15min":22.390000000000001},"min":{"1min":9.9459999999999997,"5min":7.218,"15min":7.218},"max":{"1min":22.241,"5min":136.494,"15min":213.072},"last":16.009},{"interface":"front","average":{"1min":17.73,"5min":22.32,"15min":24.960999999999999},"min":{"1min":8.9540000000000006,"5min":8.9540000000000006,"15min":8.5530000000000008},"max":{"1min":28.087,"5min":104.538,"15min":239.76900000000001},"last":16.603999999999999}]},{"osd":5,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.374000000000001,"5min":18.962,"15min":22.965},"min":{"1min":10.206,"5min":8.2889999999999997,"15min":7.258},"max":{"1min":27.122,"5min":110.45099999999999,"15min":275.08100000000002},"last":8.8789999999999996},{"interface":"front","average":{"1min":13.379,"5min":15.702999999999999,"15min":18.018000000000001},"min":{"1min":5.3410000000000002,"5min":5.3410000000000002,"15min":5.3410000000000002},"max":{"1min":40.390000000000001,"5min":94.302999999999997,"15min":129.935},"last":6.266}]},{"osd":6,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":18.283000000000001,"5min":21.891999999999999,"15min":25.209},"min":{"1min":11.725,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":27.867999999999999,"5min":126.001,"15min":263.41399999999999},"last":17.983000000000001},{"interface":"front","average":{"1min":17.962,"5min":19.896000000000001,"15min":23.355},"min":{"1min":11.305999999999999,"5min":8.0229999999999997,"15min":8.0229999999999997},"max":{"1min":56.701000000000001,"5min":111.691,"15min":257.75900000000001},"last":12.311}]}]},{"osd":2,"up_from":14,"seq":60129542414,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82076,"kb_used_data":6484,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289764,"statfs":{"total":96636764160,"available":96552718336,"internally_reserved":0,"allocated":6639616,"data_stored":3147952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":19.824999999999999,"5min":15.673,"15min":19.306999999999999},"min":{"1min":4.6959999999999997,"5min":4.3170000000000002,"15min":4.3170000000000002},"max":{"1min":211.80500000000001,"5min":211.80500000000001,"15min":389.31400000000002},"last":9.4559999999999995},{"interface":"front","average":{"1min":18.757999999999999,"5min":20.440999999999999,"15min":19.539000000000001},"min":{"1min":5.9269999999999996,"5min":5.3040000000000003,"15min":4.4589999999999996},"max":{"1min":213.93600000000001,"5min":295.87,"15min":295.87},"last":6.1929999999999996}]},{"osd":1,"last update":"Mon Oct 30 20:45:48 2023","interfaces":[{"interface":"back","average":{"1min":14.757,"5min":16.274999999999999,"15min":22.295999999999999},"min":{"1min":6.4720000000000004,"5min":5.5060000000000002,"15min":5.2649999999999997},"max":{"1min":36.689,"5min":74.575000000000003,"15min":245.84999999999999},"last":12.846},{"interface":"front","average":{"1min":17.843,"5min":15.342000000000001,"15min":20.946999999999999},"min":{"1min":7.9459999999999997,"5min":5.8399999999999999,"15min":5.3330000000000002},"max":{"1min":66.947999999999993,"5min":72.662999999999997,"15min":317.20699999999999},"last":12.295}]},{"osd":3,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.094000000000001,"5min":17.843,"15min":22.158000000000001},"min":{"1min":6.6040000000000001,"5min":6.1459999999999999,"15min":5.2619999999999996},"max":{"1min":58.442999999999998,"5min":253.02799999999999,"15min":311.03100000000001},"last":7.4980000000000002},{"interface":"front","average":{"1min":20.954000000000001,"5min":21.210999999999999,"15min":28.140999999999998},"min":{"1min":10.403,"5min":7.4969999999999999,"15min":6.2039999999999997},"max":{"1min":86.935000000000002,"5min":145.066,"15min":703},"last":10.103}]},{"osd":4,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":14.718,"5min":18.241,"15min":26.344999999999999},"min":{"1min":7.4029999999999996,"5min":6.4809999999999999,"15min":6.3460000000000001},"max":{"1min":43.838999999999999,"5min":123.06399999999999,"15min":709.85900000000004},"last":8.7910000000000004},{"interface":"front","average":{"1min":16.712,"5min":19.233000000000001,"15min":22.349},"min":{"1min":6.2359999999999998,"5min":6.2359999999999998,"15min":6.2359999999999998},"max":{"1min":65.897000000000006,"5min":118.82599999999999,"15min":182.947},"last":13.544}]},{"osd":5,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":18.765000000000001,"5min":18.805,"15min":29.754000000000001},"min":{"1min":8.3539999999999992,"5min":7.2279999999999998,"15min":6.8460000000000001},"max":{"1min":63.963000000000001,"5min":116.352,"15min":874.22799999999995},"last":11.289},{"interface":"front","average":{"1min":17.289999999999999,"5min":18.369,"15min":24.251999999999999},"min":{"1min":6.9500000000000002,"5min":6.9500000000000002,"15min":6.9500000000000002},"max":{"1min":41.155000000000001,"5min":73.218999999999994,"15min":222.76400000000001},"last":14.624000000000001}]},{"osd":6,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":15.76,"5min":23.898,"15min":25.614999999999998},"min":{"1min":8.5389999999999997,"5min":8.5389999999999997,"15min":7.3049999999999997},"max":{"1min":37.296999999999997,"5min":196.654,"15min":238.84800000000001},"last":15.263999999999999},{"interface":"front","average":{"1min":18.132000000000001,"5min":23.280000000000001,"15min":29.577000000000002},"min":{"1min":8.3770000000000007,"5min":8.0670000000000002,"15min":7.577},"max":{"1min":37.128999999999998,"5min":161.30799999999999,"15min":266.48599999999999},"last":16.105}]},{"osd":7,"last update":"Mon Oct 30 20:45:31 2023","interfaces":[{"interface":"back","average":{"1min":19.760000000000002,"5min":20.834,"15min":26.890000000000001},"min":{"1min":9.7300000000000004,"5min":9.7300000000000004,"15min":9.7300000000000004},"max":{"1min":39.747,"5min":133.429,"15min":354.904},"last":16.795000000000002},{"interface":"front","average":{"1min":19.949999999999999,"5min":21.204000000000001,"15min":29.919},"min":{"1min":9.1600000000000001,"5min":7.3440000000000003,"15min":7.3440000000000003},"max":{"1min":75.688999999999993,"5min":153.803,"15min":865.25800000000004},"last":17.919}]}]},{"osd":4,"up_from":13,"seq":55834575120,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82072,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289768,"statfs":{"total":96636764160,"available":96552722432,"internally_reserved":0,"allocated":6635520,"data_stored":3147947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":12.099,"5min":10.295,"15min":15.339},"min":{"1min":4.5940000000000003,"5min":3.8839999999999999,"15min":3.839},"max":{"1min":73.688000000000002,"5min":118.02500000000001,"15min":224.95099999999999},"last":10.99},{"interface":"front","average":{"1min":14.191000000000001,"5min":11.253,"15min":15.319000000000001},"min":{"1min":4.407,"5min":4.407,"15min":4.0229999999999997},"max":{"1min":70.055999999999997,"5min":70.055999999999997,"15min":227.71600000000001},"last":11.563000000000001}]},{"osd":1,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":19.260999999999999,"5min":15.923,"15min":20.707999999999998},"min":{"1min":6.3550000000000004,"5min":4.7960000000000003,"15min":4.7960000000000003},"max":{"1min":92.763000000000005,"5min":118.747,"15min":383.57499999999999},"last":7.7030000000000003},{"interface":"front","average":{"1min":20.491,"5min":19.440000000000001,"15min":22.885999999999999},"min":{"1min":8.3000000000000007,"5min":7.8970000000000002,"15min":7.5},"max":{"1min":59.844999999999999,"5min":110.129,"15min":228.34999999999999},"last":16.431000000000001}]},{"osd":2,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":17.518000000000001,"5min":13.864000000000001,"15min":21.329000000000001},"min":{"1min":8.2390000000000008,"5min":4.6420000000000003,"15min":4.6420000000000003},"max":{"1min":54.417999999999999,"5min":82.471999999999994,"15min":347.30799999999999},"last":9.7750000000000004},{"interface":"front","average":{"1min":23.317,"5min":16.451000000000001,"15min":21.233000000000001},"min":{"1min":6.1369999999999996,"5min":6.1369999999999996,"15min":6.0090000000000003},"max":{"1min":121.45999999999999,"5min":121.45999999999999,"15min":183.03100000000001},"last":13.837999999999999}]},{"osd":3,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":16.106000000000002,"5min":17.736000000000001,"15min":24.536000000000001},"min":{"1min":6.4779999999999998,"5min":6.4779999999999998,"15min":6.2930000000000001},"max":{"1min":70.343999999999994,"5min":174.733,"15min":270.19400000000002},"last":12.179},{"interface":"front","average":{"1min":18.103999999999999,"5min":17.949999999999999,"15min":24.756},"min":{"1min":6.0220000000000002,"5min":6.0220000000000002,"15min":6.0220000000000002},"max":{"1min":63.222000000000001,"5min":177.87,"15min":339.79700000000003},"last":17.242999999999999}]},{"osd":5,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":20.681000000000001,"5min":20.640999999999998,"15min":24.623999999999999},"min":{"1min":6.5839999999999996,"5min":6.5839999999999996,"15min":6.5839999999999996},"max":{"1min":158.541,"5min":158.541,"15min":253.80199999999999},"last":18.027000000000001},{"interface":"front","average":{"1min":17.707000000000001,"5min":16.873000000000001,"15min":22.225000000000001},"min":{"1min":6.1029999999999998,"5min":6.1029999999999998,"15min":6.1029999999999998},"max":{"1min":147.54400000000001,"5min":147.54400000000001,"15min":271.572},"last":9.0299999999999994}]},{"osd":6,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":26.273,"5min":18.638999999999999,"15min":24.821999999999999},"min":{"1min":7.0999999999999996,"5min":6.4210000000000003,"15min":6.4210000000000003},"max":{"1min":245.447,"5min":245.447,"15min":254.44399999999999},"last":6.7050000000000001},{"interface":"front","average":{"1min":18.928999999999998,"5min":19.975000000000001,"15min":28.715},"min":{"1min":10.25,"5min":9.0419999999999998,"15min":8.1639999999999997},"max":{"1min":72.599000000000004,"5min":74.463999999999999,"15min":291.62599999999998},"last":15.385999999999999}]},{"osd":7,"last update":"Mon Oct 30 20:45:36 2023","interfaces":[{"interface":"back","average":{"1min":14.513,"5min":16.260000000000002,"15min":22.474},"min":{"1min":8.1890000000000001,"5min":8.1890000000000001,"15min":6.2389999999999999},"max":{"1min":43.911999999999999,"5min":76.325000000000003,"15min":262.82100000000003},"last":10.382999999999999},{"interface":"front","average":{"1min":19.515000000000001,"5min":19.870999999999999,"15min":25.632000000000001},"min":{"1min":9.8350000000000009,"5min":9.8350000000000009,"15min":9.8350000000000009},"max":{"1min":61.091000000000001,"5min":61.091000000000001,"15min":244.37799999999999},"last":14.67}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T20:46:22.452 INFO:tasks.ceph:pgid 10.17 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:22.453 INFO:tasks.ceph:pgid 10.11 last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:22.454 INFO:tasks.ceph:pgid 10.1d last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:22.455 INFO:tasks.ceph:pgid 10.f last_scrub_stamp 2023-10-30T20:25:17.587663+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=25, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=20, tm_min=45, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T20:46:22.456 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T20:46:30.428 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:52.485 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:30.433 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:52.491 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:30.456 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:52.508 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:30.480 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:52.507 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:30.482 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:52.541 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:30.487 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:52.511 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:30.493 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:52.550 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:30.499 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:52.526 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:30.509 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:52.568 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:31.111 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:53.138 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:31.113 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:53.140 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:31.120 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:23:53.147 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:31.140 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:53.198 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:31.141 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:53.200 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:31.149 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:53.208 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:31.161 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:53.220 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:31.163 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:53.222 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:31.174 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:53.232 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:42.458 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T20:46:42.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.561+0000 7ff7bbfc9640 10 monclient: get_monmap_and_config 2023-10-30T20:46:42.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.561+0000 7ff7bbfc9640 10 monclient: build_initial_monmap 2023-10-30T20:46:42.563 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.561+0000 7ff7bbfc9640 10 monclient: monmap: 2023-10-30T20:46:42.563 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:46:42.563 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:42.563 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:46:42.562524+0000 2023-10-30T20:46:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:46:42.562524+0000 2023-10-30T20:46:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:46:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:46:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:46:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:46:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:46:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:46:42.565 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:42.567 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 1 Processor -- start 2023-10-30T20:46:42.567 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 1 -- start start 2023-10-30T20:46:42.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 10 monclient: init 2023-10-30T20:46:42.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 10 monclient: _reopen_session rank -1 2023-10-30T20:46:42.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-10-30T20:46:42.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff7b4065d20 addr v1:172.21.15.183:6789/0 2023-10-30T20:46:42.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff7b40ad120 0x7ff7b41453a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:42.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 10 monclient(hunting): picked mon.noname-c con 0x7ff7b40ad120 addr v2:172.21.15.183:3300/0 2023-10-30T20:46:42.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b41458e0 0x7ff7b414c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:42.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 10 monclient(hunting): picked mon.noname-b con 0x7ff7b41458e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:46:42.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b413f460 con 0x7ff7b4065d20 2023-10-30T20:46:42.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:42.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b4067a30 con 0x7ff7b40ad120 2023-10-30T20:46:42.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:42.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b406eb90 con 0x7ff7b41458e0 2023-10-30T20:46:42.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 10 monclient(hunting): _renew_subs 2023-10-30T20:46:42.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.562+0000 7ff7bbfc9640 10 monclient(hunting): authenticate will time out at 2925.851743s 2023-10-30T20:46:42.570 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.563+0000 7ff7ba566640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b41458e0 0x7ff7b414c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:42.571 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.563+0000 7ff7b9564640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff7b40ad120 0x7ff7b41453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:42.571 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.563+0000 7ff7ba566640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b41458e0 0x7ff7b414c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:51610/0 (socket says 172.21.15.69:51610) 2023-10-30T20:46:42.571 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.563+0000 7ff7ba566640 1 -- 172.21.15.69:0/2458917458 learned_addr learned my addr 172.21.15.69:0/2458917458 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:46:42.571 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.563+0000 7ff7ba566640 10 monclient(hunting): get_auth_request con 0x7ff7b41458e0 auth_method 0 2023-10-30T20:46:42.571 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.563+0000 7ff7ba566640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:42.572 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.563+0000 7ff7ba566640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:42.572 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.563+0000 7ff7ba566640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:42.572 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.564+0000 7ff7b9564640 10 monclient(hunting): get_auth_request con 0x7ff7b40ad120 auth_method 0 2023-10-30T20:46:42.572 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.564+0000 7ff7b9564640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:42.572 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.564+0000 7ff7b9564640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:42.572 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.564+0000 7ff7b9564640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:42.573 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.564+0000 7ff7ba566640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:42.573 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.564+0000 7ff7ba566640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:42.573 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.564+0000 7ff7ba566640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:42.573 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.565+0000 7ff7b9564640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:42.573 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.565+0000 7ff7b9564640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:42.574 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.565+0000 7ff7b9564640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:42.574 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.566+0000 7ff7ba566640 10 monclient(hunting): handle_auth_done global_id 18362 payload 274 2023-10-30T20:46:42.574 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.566+0000 7ff7ba566640 1 -- 172.21.15.69:0/2458917458 >> v2:172.21.15.183:3300/0 conn(0x7ff7b40ad120 msgr2=0x7ff7b41453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:46:42.574 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.566+0000 7ff7ba566640 1 --2- 172.21.15.69:0/2458917458 >> v2:172.21.15.183:3300/0 conn(0x7ff7b40ad120 0x7ff7b41453a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:42.574 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.566+0000 7ff7ba566640 1 -- 172.21.15.69:0/2458917458 >> v1:172.21.15.183:6789/0 conn(0x7ff7b4065d20 legacy=0x7ff7b413ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:42.574 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.566+0000 7ff7ba566640 10 monclient: _finish_hunting 0 2023-10-30T20:46:42.575 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.566+0000 7ff7ba566640 1 monclient: found mon.noname-b 2023-10-30T20:46:42.575 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.566+0000 7ff7ba566640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:46:42.575 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.566+0000 7ff7ba566640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:46:42.575 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.566+0000 7ff7ba566640 1 -- 172.21.15.69:0/2458917458 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b406ed90 con 0x7ff7b41458e0 2023-10-30T20:46:42.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.567+0000 7ff7ba566640 1 --2- 172.21.15.69:0/2458917458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b41458e0 0x7ff7b414c170 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac009e50 tx=0x7ff7ac02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddf442b9a6c3ebed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:42.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7b8d63640 1 -- 172.21.15.69:0/2458917458 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac02ca70 con 0x7ff7b41458e0 2023-10-30T20:46:42.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7b8d63640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:42.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7b8d63640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:46:42.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7b8d63640 10 monclient: dump: 2023-10-30T20:46:42.580 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:42.580 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:42.580 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:42.580 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:42.581 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:42.581 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:42.581 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:42.581 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:42.581 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:42.581 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:42.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7b8d63640 10 monclient: _finish_auth 0 2023-10-30T20:46:42.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7b8d63640 10 monclient: _check_auth_tickets 2023-10-30T20:46:42.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7b8d63640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:46:42.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7b8d63640 1 -- 172.21.15.69:0/2458917458 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ac02cbe0 con 0x7ff7b41458e0 2023-10-30T20:46:42.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7b8d63640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:46:42.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7b8d63640 1 -- 172.21.15.69:0/2458917458 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac0343f0 con 0x7ff7b41458e0 2023-10-30T20:46:42.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7b8d63640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:42.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7b8d63640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:46:42.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7b8d63640 10 monclient: dump: 2023-10-30T20:46:42.583 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:42.583 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:42.584 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:42.584 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:42.584 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:42.584 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:42.584 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:42.585 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:42.585 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:42.585 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:42.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7bbfc9640 5 monclient: authenticate success, global_id 18362 2023-10-30T20:46:42.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7bbfc9640 10 monclient: get_monmap_and_config success 2023-10-30T20:46:42.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.571+0000 7ff7bbfc9640 4 set_mon_vals no callback set 2023-10-30T20:46:42.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.572+0000 7ff7bbfc9640 10 monclient: shutdown 2023-10-30T20:46:42.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.572+0000 7ff7bbfc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:46:42.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.572+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/2458917458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b41458e0 msgr2=0x7ff7b414c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:42.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.572+0000 7ff7bbfc9640 1 --2- 172.21.15.69:0/2458917458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b41458e0 0x7ff7b414c170 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac009e50 tx=0x7ff7ac02bed0 comp rx=0 tx=0).stop 2023-10-30T20:46:42.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.572+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/2458917458 shutdown_connections 2023-10-30T20:46:42.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.572+0000 7ff7bbfc9640 1 --2- 172.21.15.69:0/2458917458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b41458e0 0x7ff7b414c170 unknown :-1 s=CLOSED pgs=3222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:42.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.572+0000 7ff7bbfc9640 1 --2- 172.21.15.69:0/2458917458 >> v2:172.21.15.183:3300/0 conn(0x7ff7b40ad120 0x7ff7b41453a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:42.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.572+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/2458917458 >> 172.21.15.69:0/2458917458 conn(0x7ff7b40b3c00 msgr2=0x7ff7b40b0050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:46:42.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.572+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/2458917458 shutdown_connections 2023-10-30T20:46:42.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.572+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/2458917458 wait complete. 2023-10-30T20:46:42.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.572+0000 7ff7bbfc9640 10 monclient: build_initial_monmap 2023-10-30T20:46:42.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.572+0000 7ff7bbfc9640 10 monclient: monmap: 2023-10-30T20:46:42.588 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:46:42.588 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:46:42.588 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:46:42.588 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:46:42.588 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:46:42.588 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:46:42.589 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:46:42.589 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:46:42.589 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:46:42.589 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:42.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 1 librados: starting msgr at 2023-10-30T20:46:42.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 1 librados: starting objecter 2023-10-30T20:46:42.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 1 Processor -- start 2023-10-30T20:46:42.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 1 -- start start 2023-10-30T20:46:42.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 1 librados: setting wanted keys 2023-10-30T20:46:42.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 1 librados: calling monclient init 2023-10-30T20:46:42.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 10 monclient: init 2023-10-30T20:46:42.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 10 monclient: _reopen_session rank -1 2023-10-30T20:46:42.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T20:46:42.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b4065d20 0x7ff7b41415e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:42.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 10 monclient(hunting): picked mon.noname-c con 0x7ff7b4065d20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:46:42.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b40ad120 0x7ff7b4143900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:42.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 10 monclient(hunting): picked mon.noname-b con 0x7ff7b40ad120 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:46:42.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b41458e0 0x7ff7b4140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:42.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 10 monclient(hunting): picked mon.noname-a con 0x7ff7b41458e0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:46:42.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:42.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b4142050 con 0x7ff7b41458e0 2023-10-30T20:46:42.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:42.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b406fbc0 con 0x7ff7b40ad120 2023-10-30T20:46:42.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:42.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b406f6a0 con 0x7ff7b4065d20 2023-10-30T20:46:42.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 10 monclient(hunting): _renew_subs 2023-10-30T20:46:42.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.573+0000 7ff7bbfc9640 10 monclient(hunting): authenticate will time out at 2925.862704s 2023-10-30T20:46:42.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.574+0000 7ff7b9d65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b4065d20 0x7ff7b41415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:42.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.574+0000 7ff7ba566640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b41458e0 0x7ff7b4140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:42.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.574+0000 7ff7b9564640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b40ad120 0x7ff7b4143900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:42.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.574+0000 7ff7b9d65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b4065d20 0x7ff7b41415e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:51624/0 (socket says 172.21.15.69:51624) 2023-10-30T20:46:42.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.574+0000 7ff7b9d65640 1 -- 172.21.15.69:0/4286459634 learned_addr learned my addr 172.21.15.69:0/4286459634 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:46:42.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.574+0000 7ff7b9d65640 10 monclient(hunting): get_auth_request con 0x7ff7b4065d20 auth_method 0 2023-10-30T20:46:42.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.574+0000 7ff7b9d65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:42.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.574+0000 7ff7b9d65640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:42.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.574+0000 7ff7b9d65640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:42.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.574+0000 7ff7ba566640 10 monclient(hunting): get_auth_request con 0x7ff7b41458e0 auth_method 0 2023-10-30T20:46:42.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.574+0000 7ff7ba566640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:42.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.574+0000 7ff7ba566640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:42.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.574+0000 7ff7ba566640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:42.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.575+0000 7ff7b9564640 10 monclient(hunting): get_auth_request con 0x7ff7b40ad120 auth_method 0 2023-10-30T20:46:42.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.575+0000 7ff7b9564640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:42.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.575+0000 7ff7b9564640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:42.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.575+0000 7ff7b9564640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:42.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.575+0000 7ff7b9d65640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:42.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.575+0000 7ff7b9d65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:42.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.575+0000 7ff7b9d65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:42.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.575+0000 7ff7ba566640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:42.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.575+0000 7ff7ba566640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:42.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.575+0000 7ff7ba566640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:42.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.575+0000 7ff7b9564640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:42.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.575+0000 7ff7b9564640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:42.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.575+0000 7ff7b9564640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:42.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.577+0000 7ff7b9d65640 10 monclient(hunting): handle_auth_done global_id 18365 payload 824 2023-10-30T20:46:42.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.577+0000 7ff7b9d65640 1 -- 172.21.15.69:0/4286459634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b40ad120 msgr2=0x7ff7b4143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:42.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.577+0000 7ff7b9d65640 1 --2- 172.21.15.69:0/4286459634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b40ad120 0x7ff7b4143900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:42.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.577+0000 7ff7b9d65640 1 -- 172.21.15.69:0/4286459634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b41458e0 msgr2=0x7ff7b4140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:42.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.577+0000 7ff7b9d65640 1 --2- 172.21.15.69:0/4286459634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b41458e0 0x7ff7b4140ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:42.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.577+0000 7ff7b9d65640 10 monclient: _finish_hunting 0 2023-10-30T20:46:42.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.577+0000 7ff7b9d65640 1 monclient: found mon.noname-c 2023-10-30T20:46:42.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.577+0000 7ff7b9d65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:46:42.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.577+0000 7ff7b9d65640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:46:42.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.577+0000 7ff7b9d65640 1 -- 172.21.15.69:0/4286459634 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b41d65b0 con 0x7ff7b4065d20 2023-10-30T20:46:42.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.579+0000 7ff7b9d65640 1 --2- 172.21.15.69:0/4286459634 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b4065d20 0x7ff7b41415e0 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 crypto rx=0x7ff7b000c480 tx=0x7ff7b000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:42.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.579+0000 7ff7aa7fc640 10 client.?.objecter ms_handle_connect 0x7ff7b4065d20 2023-10-30T20:46:42.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.579+0000 7ff7aa7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T20:46:42.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7aa7fc640 1 -- 172.21.15.69:0/4286459634 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b00052d0 con 0x7ff7b4065d20 2023-10-30T20:46:42.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:42.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7aa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:46:42.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7aa7fc640 10 monclient: dump: 2023-10-30T20:46:42.603 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:42.603 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:42.603 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:42.603 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:42.603 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:42.604 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:42.604 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:42.604 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:42.604 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:42.604 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:42.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7aa7fc640 10 monclient: _finish_auth 0 2023-10-30T20:46:42.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7aa7fc640 10 monclient: _check_auth_tickets 2023-10-30T20:46:42.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:46:42.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7aa7fc640 1 -- 172.21.15.69:0/4286459634 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b000e040 con 0x7ff7b4065d20 2023-10-30T20:46:42.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7bbfc9640 5 monclient: authenticate success, global_id 18365 2023-10-30T20:46:42.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7bbfc9640 10 monclient: _renew_subs 2023-10-30T20:46:42.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7bbfc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:46:42.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/4286459634 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b41d6870 con 0x7ff7b4065d20 2023-10-30T20:46:42.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7aa7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:46:42.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7aa7fc640 1 -- 172.21.15.69:0/4286459634 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0012630 con 0x7ff7b4065d20 2023-10-30T20:46:42.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:42.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:46:42.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7aa7fc640 10 monclient: dump: 2023-10-30T20:46:42.607 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:42.607 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:42.607 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:42.607 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:42.608 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:42.608 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:42.608 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:42.608 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:42.608 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:42.609 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:42.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7abfff640 4 set_mon_vals no callback set 2023-10-30T20:46:42.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7bbfc9640 10 client.18365.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:46:42.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7bbfc9640 10 monclient: _renew_subs 2023-10-30T20:46:42.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7bbfc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:46:42.609 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/4286459634 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b41d6d60 con 0x7ff7b4065d20 2023-10-30T20:46:42.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.583+0000 7ff7bbfc9640 1 librados: init done 2023-10-30T20:46:42.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.584+0000 7ff7bbfc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:42.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.584+0000 7ff7bbfc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:42.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.584+0000 7ff7bbfc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:46:42.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.584+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/4286459634 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b404e9b0 con 0x7ff7b4065d20 2023-10-30T20:46:42.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.605+0000 7ff7aa7fc640 1 -- 172.21.15.69:0/4286459634 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7ff7b00127a0 con 0x7ff7b4065d20 2023-10-30T20:46:42.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.606+0000 7ff7aa7fc640 1 --2- 172.21.15.69:0/4286459634 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff79806c810 0x7ff79806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:42.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.606+0000 7ff7b9564640 1 --2- 172.21.15.69:0/4286459634 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff79806c810 0x7ff79806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:42.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.606+0000 7ff7b9564640 10 monclient: get_auth_request con 0x7ff79806c810 auth_method 0 2023-10-30T20:46:42.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.606+0000 7ff7aa7fc640 10 client.18365.objecter ms_handle_connect 0x7ff79806c810 2023-10-30T20:46:42.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.606+0000 7ff7b9564640 1 --2- 172.21.15.69:0/4286459634 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff79806c810 0x7ff79806ecd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4143630 tx=0x7ff7a4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:42.838 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.836+0000 7ff7aa7fc640 1 -- 172.21.15.69:0/4286459634 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7ff7b0058350 con 0x7ff7b4065d20 2023-10-30T20:46:42.838 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.836+0000 7ff7aa7fc640 10 client.18365.objecter ms_dispatch 0x7ff7b4000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:46:42.838 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.836+0000 7ff7aa7fc640 3 client.18365.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:46:42.838 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.836+0000 7ff7aa7fc640 3 client.18365.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:46:42.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.836+0000 7ff7aa7fc640 20 client.18365.objecter dump_active .. 0 homeless 2023-10-30T20:46:42.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.836+0000 7ff7aa7fc640 1 -- 172.21.15.69:0/4286459634 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff7b005c3f0 con 0x7ff7b4065d20 2023-10-30T20:46:42.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.836+0000 7ff7aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:42.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:42.836+0000 7ff7aa7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:46:43.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.035+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/4286459634 --> v2:172.21.15.69:6800/1876307353 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff7b4066100 con 0x7ff79806c810 2023-10-30T20:46:43.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.039+0000 7ff7aa7fc640 1 -- 172.21.15.69:0/4286459634 <== mgr.4105 v2:172.21.15.69:6800/1876307353 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152077 (secure 0 0 0) 0x7ff7a4011a30 con 0x7ff79806c810 2023-10-30T20:46:43.042 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-30T20:46:43.043 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-10-30T20:46:43.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.041+0000 7ff7bbfc9640 10 librados: watch_flush enter 2023-10-30T20:46:43.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 10 librados: watch_flush exit 2023-10-30T20:46:43.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 20 client.18365.objecter shutdown clearing up homeless session... 2023-10-30T20:46:43.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 10 client.18365.objecter successfully canceled tick 2023-10-30T20:46:43.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/4286459634 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff79806c810 msgr2=0x7ff79806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:43.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 1 --2- 172.21.15.69:0/4286459634 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff79806c810 0x7ff79806ecd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4143630 tx=0x7ff7a4009190 comp rx=0 tx=0).stop 2023-10-30T20:46:43.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 10 monclient: shutdown 2023-10-30T20:46:43.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:46:43.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/4286459634 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b4065d20 msgr2=0x7ff7b41415e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:43.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 1 --2- 172.21.15.69:0/4286459634 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b4065d20 0x7ff7b41415e0 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 crypto rx=0x7ff7b000c480 tx=0x7ff7b000c940 comp rx=0 tx=0).stop 2023-10-30T20:46:43.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/4286459634 shutdown_connections 2023-10-30T20:46:43.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 1 --2- 172.21.15.69:0/4286459634 >> v2:172.21.15.69:6800/1876307353 conn(0x7ff79806c810 0x7ff79806ecd0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:43.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 1 --2- 172.21.15.69:0/4286459634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b41458e0 0x7ff7b4140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:43.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 1 --2- 172.21.15.69:0/4286459634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b40ad120 0x7ff7b4143900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:43.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 1 --2- 172.21.15.69:0/4286459634 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b4065d20 0x7ff7b41415e0 unknown :-1 s=CLOSED pgs=3223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:43.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/4286459634 >> 172.21.15.69:0/4286459634 conn(0x7ff7b40b3c00 msgr2=0x7ff7b413eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:46:43.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/4286459634 shutdown_connections 2023-10-30T20:46:43.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 1 -- 172.21.15.69:0/4286459634 wait complete. 2023-10-30T20:46:43.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.042+0000 7ff7bbfc9640 1 librados: shutdown 2023-10-30T20:46:43.054 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1593,"stamp":"2023-10-30T20:46:42.513355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":545616,"kb_used_data":51920,"kb_used_omap":12,"kb_used_meta":493619,"kb_avail":754429104,"statfs":{"total":773094113280,"available":772535402496,"internally_reserved":0,"allocated":53166080,"data_stored":25979266,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12745,"internal_metadata":505466423},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002758"},"pg_stats":[{"pgid":"10.16","version":"0'0","reported_seq":856,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:57.478854+0000","last_change":"2023-10-30T20:45:57.479913+0000","last_active":"2023-10-30T20:45:57.478854+0000","last_peered":"2023-10-30T20:45:57.478854+0000","last_clean":"2023-10-30T20:45:57.478854+0000","last_became_active":"2023-10-30T20:25:23.962078+0000","last_became_peered":"2023-10-30T20:25:23.962078+0000","last_unstale":"2023-10-30T20:45:57.478854+0000","last_undegraded":"2023-10-30T20:45:57.478854+0000","last_fullsized":"2023-10-30T20:45:57.478854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:57.477507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:57.477507+0000","last_clean_scrub_stamp":"2023-10-30T20:45:57.477507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:20:59.025555+0000","scrub_duration":0.036447980999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":880,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:20.472672+0000","last_change":"2023-10-30T20:46:20.476312+0000","last_active":"2023-10-30T20:46:20.472672+0000","last_peered":"2023-10-30T20:46:20.472672+0000","last_clean":"2023-10-30T20:46:20.472672+0000","last_became_active":"2023-10-30T20:25:24.861824+0000","last_became_peered":"2023-10-30T20:25:24.861824+0000","last_unstale":"2023-10-30T20:46:20.472672+0000","last_undegraded":"2023-10-30T20:46:20.472672+0000","last_fullsized":"2023-10-30T20:46:20.472672+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:20.471347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:20.471347+0000","last_clean_scrub_stamp":"2023-10-30T20:46:20.471347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:17:41.228524+0000","scrub_duration":0.071503348999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":862,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:04.228990+0000","last_change":"2023-10-30T20:46:04.228990+0000","last_active":"2023-10-30T20:46:04.228990+0000","last_peered":"2023-10-30T20:46:04.228990+0000","last_clean":"2023-10-30T20:46:04.228990+0000","last_became_active":"2023-10-30T20:25:36.951760+0000","last_became_peered":"2023-10-30T20:25:36.951760+0000","last_unstale":"2023-10-30T20:46:04.228990+0000","last_undegraded":"2023-10-30T20:46:04.228990+0000","last_fullsized":"2023-10-30T20:46:04.228990+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:04.226007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:04.226007+0000","last_clean_scrub_stamp":"2023-10-30T20:46:04.226007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:49:58.849033+0000","scrub_duration":0.052304344000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":878,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:12.909228+0000","last_change":"2023-10-30T20:46:12.910204+0000","last_active":"2023-10-30T20:46:12.909228+0000","last_peered":"2023-10-30T20:46:12.909228+0000","last_clean":"2023-10-30T20:46:12.909228+0000","last_became_active":"2023-10-30T20:25:23.356998+0000","last_became_peered":"2023-10-30T20:25:23.356998+0000","last_unstale":"2023-10-30T20:46:12.909228+0000","last_undegraded":"2023-10-30T20:46:12.909228+0000","last_fullsized":"2023-10-30T20:46:12.909228+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:12.907859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:12.907859+0000","last_clean_scrub_stamp":"2023-10-30T20:46:12.907859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:12:47.708802+0000","scrub_duration":0.042919487999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":890,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:16.847330+0000","last_change":"2023-10-30T20:46:16.847330+0000","last_active":"2023-10-30T20:46:16.847330+0000","last_peered":"2023-10-30T20:46:16.847330+0000","last_clean":"2023-10-30T20:46:16.847330+0000","last_became_active":"2023-10-30T20:25:25.477142+0000","last_became_peered":"2023-10-30T20:25:25.477142+0000","last_unstale":"2023-10-30T20:46:16.847330+0000","last_undegraded":"2023-10-30T20:46:16.847330+0000","last_fullsized":"2023-10-30T20:46:16.847330+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:16.844987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:16.844987+0000","last_clean_scrub_stamp":"2023-10-30T20:46:16.844987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:14:32.688895+0000","scrub_duration":0.042022205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":862,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:02.185985+0000","last_change":"2023-10-30T20:46:02.185985+0000","last_active":"2023-10-30T20:46:02.185985+0000","last_peered":"2023-10-30T20:46:02.185985+0000","last_clean":"2023-10-30T20:46:02.185985+0000","last_became_active":"2023-10-30T20:25:23.840599+0000","last_became_peered":"2023-10-30T20:25:23.840599+0000","last_unstale":"2023-10-30T20:46:02.185985+0000","last_undegraded":"2023-10-30T20:46:02.185985+0000","last_fullsized":"2023-10-30T20:46:02.185985+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:02.183565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:02.183565+0000","last_clean_scrub_stamp":"2023-10-30T20:46:02.183565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:23:34.216637+0000","scrub_duration":0.065721632000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":890,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:01.765247+0000","last_change":"2023-10-30T20:46:01.765247+0000","last_active":"2023-10-30T20:46:01.765247+0000","last_peered":"2023-10-30T20:46:01.765247+0000","last_clean":"2023-10-30T20:46:01.765247+0000","last_became_active":"2023-10-30T20:25:28.691162+0000","last_became_peered":"2023-10-30T20:25:28.691162+0000","last_unstale":"2023-10-30T20:46:01.765247+0000","last_undegraded":"2023-10-30T20:46:01.765247+0000","last_fullsized":"2023-10-30T20:46:01.765247+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:01.759908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:01.759908+0000","last_clean_scrub_stamp":"2023-10-30T20:46:01.759908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:58.795323+0000","scrub_duration":0.054680237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":890,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:21.816303+0000","last_change":"2023-10-30T20:46:21.817327+0000","last_active":"2023-10-30T20:46:21.816303+0000","last_peered":"2023-10-30T20:46:21.816303+0000","last_clean":"2023-10-30T20:46:21.816303+0000","last_became_active":"2023-10-30T20:25:36.286902+0000","last_became_peered":"2023-10-30T20:25:36.286902+0000","last_unstale":"2023-10-30T20:46:21.816303+0000","last_undegraded":"2023-10-30T20:46:21.816303+0000","last_fullsized":"2023-10-30T20:46:21.816303+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:21.814918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:21.814918+0000","last_clean_scrub_stamp":"2023-10-30T20:46:21.814918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:56:52.011632+0000","scrub_duration":0.054805999000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":867,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:03.143318+0000","last_change":"2023-10-30T20:46:03.144344+0000","last_active":"2023-10-30T20:46:03.143318+0000","last_peered":"2023-10-30T20:46:03.143318+0000","last_clean":"2023-10-30T20:46:03.143318+0000","last_became_active":"2023-10-30T20:25:25.367753+0000","last_became_peered":"2023-10-30T20:25:25.367753+0000","last_unstale":"2023-10-30T20:46:03.143318+0000","last_undegraded":"2023-10-30T20:46:03.143318+0000","last_fullsized":"2023-10-30T20:46:03.143318+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:03.141961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:03.141961+0000","last_clean_scrub_stamp":"2023-10-30T20:46:03.141961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:54:33.638458+0000","scrub_duration":0.062460834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":866,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:06.364710+0000","last_change":"2023-10-30T20:46:06.364710+0000","last_active":"2023-10-30T20:46:06.364710+0000","last_peered":"2023-10-30T20:46:06.364710+0000","last_clean":"2023-10-30T20:46:06.364710+0000","last_became_active":"2023-10-30T20:25:33.362871+0000","last_became_peered":"2023-10-30T20:25:33.362871+0000","last_unstale":"2023-10-30T20:46:06.364710+0000","last_undegraded":"2023-10-30T20:46:06.364710+0000","last_fullsized":"2023-10-30T20:46:06.364710+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:06.362349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:06.362349+0000","last_clean_scrub_stamp":"2023-10-30T20:46:06.362349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:18.646609+0000","scrub_duration":0.064646558000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":866,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:07.344656+0000","last_change":"2023-10-30T20:46:07.345677+0000","last_active":"2023-10-30T20:46:07.344656+0000","last_peered":"2023-10-30T20:46:07.344656+0000","last_clean":"2023-10-30T20:46:07.344656+0000","last_became_active":"2023-10-30T20:25:35.047116+0000","last_became_peered":"2023-10-30T20:25:35.047116+0000","last_unstale":"2023-10-30T20:46:07.344656+0000","last_undegraded":"2023-10-30T20:46:07.344656+0000","last_fullsized":"2023-10-30T20:46:07.344656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:07.343285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:07.343285+0000","last_clean_scrub_stamp":"2023-10-30T20:46:07.343285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:49:32.952530+0000","scrub_duration":0.038919666999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":880,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:16.542477+0000","last_change":"2023-10-30T20:46:16.543451+0000","last_active":"2023-10-30T20:46:16.542477+0000","last_peered":"2023-10-30T20:46:16.542477+0000","last_clean":"2023-10-30T20:46:16.542477+0000","last_became_active":"2023-10-30T20:25:45.271565+0000","last_became_peered":"2023-10-30T20:25:45.271565+0000","last_unstale":"2023-10-30T20:46:16.542477+0000","last_undegraded":"2023-10-30T20:46:16.542477+0000","last_fullsized":"2023-10-30T20:46:16.542477+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:16.541124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:16.541124+0000","last_clean_scrub_stamp":"2023-10-30T20:46:16.541124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T06:41:33.948252+0000","scrub_duration":0.036889193000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":881,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:18.478414+0000","last_change":"2023-10-30T20:46:18.479426+0000","last_active":"2023-10-30T20:46:18.478414+0000","last_peered":"2023-10-30T20:46:18.478414+0000","last_clean":"2023-10-30T20:46:18.478414+0000","last_became_active":"2023-10-30T20:25:28.829991+0000","last_became_peered":"2023-10-30T20:25:28.829991+0000","last_unstale":"2023-10-30T20:46:18.478414+0000","last_undegraded":"2023-10-30T20:46:18.478414+0000","last_fullsized":"2023-10-30T20:46:18.478414+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:18.477028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:18.477028+0000","last_clean_scrub_stamp":"2023-10-30T20:46:18.477028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:08:41.425297+0000","scrub_duration":0.063824697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":886,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:59.550815+0000","last_change":"2023-10-30T20:45:59.551856+0000","last_active":"2023-10-30T20:45:59.550815+0000","last_peered":"2023-10-30T20:45:59.550815+0000","last_clean":"2023-10-30T20:45:59.550815+0000","last_became_active":"2023-10-30T20:24:04.692769+0000","last_became_peered":"2023-10-30T20:24:04.692769+0000","last_unstale":"2023-10-30T20:45:59.550815+0000","last_undegraded":"2023-10-30T20:45:59.550815+0000","last_fullsized":"2023-10-30T20:45:59.550815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:59.549436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:59.549436+0000","last_clean_scrub_stamp":"2023-10-30T20:45:59.549436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T06:18:24.134116+0000","scrub_duration":0.076865441000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":857,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:58.476474+0000","last_change":"2023-10-30T20:45:58.477487+0000","last_active":"2023-10-30T20:45:58.476474+0000","last_peered":"2023-10-30T20:45:58.476474+0000","last_clean":"2023-10-30T20:45:58.476474+0000","last_became_active":"2023-10-30T20:25:27.291944+0000","last_became_peered":"2023-10-30T20:25:27.291944+0000","last_unstale":"2023-10-30T20:45:58.476474+0000","last_undegraded":"2023-10-30T20:45:58.476474+0000","last_fullsized":"2023-10-30T20:45:58.476474+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:58.475059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:58.475059+0000","last_clean_scrub_stamp":"2023-10-30T20:45:58.475059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:37:41.642031+0000","scrub_duration":0.045242620999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":884,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:59.837045+0000","last_change":"2023-10-30T20:45:59.838090+0000","last_active":"2023-10-30T20:45:59.837045+0000","last_peered":"2023-10-30T20:45:59.837045+0000","last_clean":"2023-10-30T20:45:59.837045+0000","last_became_active":"2023-10-30T20:24:04.712006+0000","last_became_peered":"2023-10-30T20:24:04.712006+0000","last_unstale":"2023-10-30T20:45:59.837045+0000","last_undegraded":"2023-10-30T20:45:59.837045+0000","last_fullsized":"2023-10-30T20:45:59.837045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:59.835644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:59.835644+0000","last_clean_scrub_stamp":"2023-10-30T20:45:59.835644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:21:20.761019+0000","scrub_duration":0.035908962000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":858,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:05.604073+0000","last_change":"2023-10-30T20:46:05.605032+0000","last_active":"2023-10-30T20:46:05.604073+0000","last_peered":"2023-10-30T20:46:05.604073+0000","last_clean":"2023-10-30T20:46:05.604073+0000","last_became_active":"2023-10-30T20:25:39.852846+0000","last_became_peered":"2023-10-30T20:25:39.852846+0000","last_unstale":"2023-10-30T20:46:05.604073+0000","last_undegraded":"2023-10-30T20:46:05.604073+0000","last_fullsized":"2023-10-30T20:46:05.604073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:05.602716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:05.602716+0000","last_clean_scrub_stamp":"2023-10-30T20:46:05.602716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:09:03.196298+0000","scrub_duration":0.048731882999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":886,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:07.540990+0000","last_change":"2023-10-30T20:46:07.542035+0000","last_active":"2023-10-30T20:46:07.540990+0000","last_peered":"2023-10-30T20:46:07.540990+0000","last_clean":"2023-10-30T20:46:07.540990+0000","last_became_active":"2023-10-30T20:24:04.310065+0000","last_became_peered":"2023-10-30T20:24:04.310065+0000","last_unstale":"2023-10-30T20:46:07.540990+0000","last_undegraded":"2023-10-30T20:46:07.540990+0000","last_fullsized":"2023-10-30T20:46:07.540990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:07.539599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:07.539599+0000","last_clean_scrub_stamp":"2023-10-30T20:46:07.539599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:21:19.638528+0000","scrub_duration":0.034526423000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":867,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:01.356922+0000","last_change":"2023-10-30T20:46:01.357982+0000","last_active":"2023-10-30T20:46:01.356922+0000","last_peered":"2023-10-30T20:46:01.356922+0000","last_clean":"2023-10-30T20:46:01.356922+0000","last_became_active":"2023-10-30T20:25:42.322505+0000","last_became_peered":"2023-10-30T20:25:42.322505+0000","last_unstale":"2023-10-30T20:46:01.356922+0000","last_undegraded":"2023-10-30T20:46:01.356922+0000","last_fullsized":"2023-10-30T20:46:01.356922+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:01.355555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:01.355555+0000","last_clean_scrub_stamp":"2023-10-30T20:46:01.355555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:24:13.843581+0000","scrub_duration":0.035547902999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":871,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:00.012998+0000","last_change":"2023-10-30T20:46:00.014043+0000","last_active":"2023-10-30T20:46:00.012998+0000","last_peered":"2023-10-30T20:46:00.012998+0000","last_clean":"2023-10-30T20:46:00.012998+0000","last_became_active":"2023-10-30T20:24:04.525673+0000","last_became_peered":"2023-10-30T20:24:04.525673+0000","last_unstale":"2023-10-30T20:46:00.012998+0000","last_undegraded":"2023-10-30T20:46:00.012998+0000","last_fullsized":"2023-10-30T20:46:00.012998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:00.011531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:00.011531+0000","last_clean_scrub_stamp":"2023-10-30T20:46:00.011531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:55:31.626108+0000","scrub_duration":0.034773421999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":867,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:00.336217+0000","last_change":"2023-10-30T20:46:00.338214+0000","last_active":"2023-10-30T20:46:00.336217+0000","last_peered":"2023-10-30T20:46:00.336217+0000","last_clean":"2023-10-30T20:46:00.336217+0000","last_became_active":"2023-10-30T20:25:33.242618+0000","last_became_peered":"2023-10-30T20:25:33.242618+0000","last_unstale":"2023-10-30T20:46:00.336217+0000","last_undegraded":"2023-10-30T20:46:00.336217+0000","last_fullsized":"2023-10-30T20:46:00.336217+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:00.329774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:00.329774+0000","last_clean_scrub_stamp":"2023-10-30T20:46:00.329774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:30:04.958207+0000","scrub_duration":0.033263122999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":886,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:02.508415+0000","last_change":"2023-10-30T20:46:02.510511+0000","last_active":"2023-10-30T20:46:02.508415+0000","last_peered":"2023-10-30T20:46:02.508415+0000","last_clean":"2023-10-30T20:46:02.508415+0000","last_became_active":"2023-10-30T20:24:04.512341+0000","last_became_peered":"2023-10-30T20:24:04.512341+0000","last_unstale":"2023-10-30T20:46:02.508415+0000","last_undegraded":"2023-10-30T20:46:02.508415+0000","last_fullsized":"2023-10-30T20:46:02.508415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:02.504542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:02.504542+0000","last_clean_scrub_stamp":"2023-10-30T20:46:02.504542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:17.334919+0000","scrub_duration":0.031536596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":858,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:59.517184+0000","last_change":"2023-10-30T20:45:59.521395+0000","last_active":"2023-10-30T20:45:59.517184+0000","last_peered":"2023-10-30T20:45:59.517184+0000","last_clean":"2023-10-30T20:45:59.517184+0000","last_became_active":"2023-10-30T20:25:25.151444+0000","last_became_peered":"2023-10-30T20:25:25.151444+0000","last_unstale":"2023-10-30T20:45:59.517184+0000","last_undegraded":"2023-10-30T20:45:59.517184+0000","last_fullsized":"2023-10-30T20:45:59.517184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:59.515798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:59.515798+0000","last_clean_scrub_stamp":"2023-10-30T20:45:59.515798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:50:26.591499+0000","scrub_duration":0.057961194000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":883,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:04.891262+0000","last_change":"2023-10-30T20:46:04.897625+0000","last_active":"2023-10-30T20:46:04.891262+0000","last_peered":"2023-10-30T20:46:04.891262+0000","last_clean":"2023-10-30T20:46:04.891262+0000","last_became_active":"2023-10-30T20:24:05.475394+0000","last_became_peered":"2023-10-30T20:24:05.475394+0000","last_unstale":"2023-10-30T20:46:04.891262+0000","last_undegraded":"2023-10-30T20:46:04.891262+0000","last_fullsized":"2023-10-30T20:46:04.891262+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:04.887537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:04.887537+0000","last_clean_scrub_stamp":"2023-10-30T20:46:04.887537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:53:22.116720+0000","scrub_duration":0.195897509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":866,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:01.045323+0000","last_change":"2023-10-30T20:46:01.046349+0000","last_active":"2023-10-30T20:46:01.045323+0000","last_peered":"2023-10-30T20:46:01.045323+0000","last_clean":"2023-10-30T20:46:01.045323+0000","last_became_active":"2023-10-30T20:25:23.473157+0000","last_became_peered":"2023-10-30T20:25:23.473157+0000","last_unstale":"2023-10-30T20:46:01.045323+0000","last_undegraded":"2023-10-30T20:46:01.045323+0000","last_fullsized":"2023-10-30T20:46:01.045323+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:01.043950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:01.043950+0000","last_clean_scrub_stamp":"2023-10-30T20:46:01.043950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:51:31.998471+0000","scrub_duration":0.031882675999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":887,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:04.526972+0000","last_change":"2023-10-30T20:46:04.526972+0000","last_active":"2023-10-30T20:46:04.526972+0000","last_peered":"2023-10-30T20:46:04.526972+0000","last_clean":"2023-10-30T20:46:04.526972+0000","last_became_active":"2023-10-30T20:24:04.254051+0000","last_became_peered":"2023-10-30T20:24:04.254051+0000","last_unstale":"2023-10-30T20:46:04.526972+0000","last_undegraded":"2023-10-30T20:46:04.526972+0000","last_fullsized":"2023-10-30T20:46:04.526972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:04.524295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:04.524295+0000","last_clean_scrub_stamp":"2023-10-30T20:46:04.524295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T08:09:25.634732+0000","scrub_duration":0.029218244000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":890,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:02.733132+0000","last_change":"2023-10-30T20:46:02.736934+0000","last_active":"2023-10-30T20:46:02.733132+0000","last_peered":"2023-10-30T20:46:02.733132+0000","last_clean":"2023-10-30T20:46:02.733132+0000","last_became_active":"2023-10-30T20:25:36.260481+0000","last_became_peered":"2023-10-30T20:25:36.260481+0000","last_unstale":"2023-10-30T20:46:02.733132+0000","last_undegraded":"2023-10-30T20:46:02.733132+0000","last_fullsized":"2023-10-30T20:46:02.733132+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:02.731814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:02.731814+0000","last_clean_scrub_stamp":"2023-10-30T20:46:02.731814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:03:02.181324+0000","scrub_duration":0.056385602999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":886,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:58.542694+0000","last_change":"2023-10-30T20:45:58.543769+0000","last_active":"2023-10-30T20:45:58.542694+0000","last_peered":"2023-10-30T20:45:58.542694+0000","last_clean":"2023-10-30T20:45:58.542694+0000","last_became_active":"2023-10-30T20:24:04.671790+0000","last_became_peered":"2023-10-30T20:24:04.671790+0000","last_unstale":"2023-10-30T20:45:58.542694+0000","last_undegraded":"2023-10-30T20:45:58.542694+0000","last_fullsized":"2023-10-30T20:45:58.542694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:58.541256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:58.541256+0000","last_clean_scrub_stamp":"2023-10-30T20:45:58.541256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:47:21.915073+0000","scrub_duration":0.034256602999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"904'78","reported_seq":1004,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:59.822887+0000","last_change":"2023-10-30T20:45:59.822887+0000","last_active":"2023-10-30T20:45:59.822887+0000","last_peered":"2023-10-30T20:45:59.822887+0000","last_clean":"2023-10-30T20:45:59.822887+0000","last_became_active":"2023-10-30T20:23:57.098872+0000","last_became_peered":"2023-10-30T20:23:57.098872+0000","last_unstale":"2023-10-30T20:45:59.822887+0000","last_undegraded":"2023-10-30T20:45:59.822887+0000","last_fullsized":"2023-10-30T20:45:59.822887+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'78","last_scrub_stamp":"2023-10-30T20:45:59.817552+0000","last_deep_scrub":"904'78","last_deep_scrub_stamp":"2023-10-30T20:45:59.817552+0000","last_clean_scrub_stamp":"2023-10-30T20:45:59.817552+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:18:28.510123+0000","scrub_duration":0.095846543000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"904'8","reported_seq":873,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:02.565655+0000","last_change":"2023-10-30T20:46:02.565655+0000","last_active":"2023-10-30T20:46:02.565655+0000","last_peered":"2023-10-30T20:46:02.565655+0000","last_clean":"2023-10-30T20:46:02.565655+0000","last_became_active":"2023-10-30T20:25:25.362636+0000","last_became_peered":"2023-10-30T20:25:25.362636+0000","last_unstale":"2023-10-30T20:46:02.565655+0000","last_undegraded":"2023-10-30T20:46:02.565655+0000","last_fullsized":"2023-10-30T20:46:02.565655+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'8","last_scrub_stamp":"2023-10-30T20:46:02.560805+0000","last_deep_scrub":"904'8","last_deep_scrub_stamp":"2023-10-30T20:46:02.560805+0000","last_clean_scrub_stamp":"2023-10-30T20:46:02.560805+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:39.733451+0000","scrub_duration":0.073184763999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":868,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:45:59.300374+0000","last_change":"2023-10-30T20:45:59.302082+0000","last_active":"2023-10-30T20:45:59.300374+0000","last_peered":"2023-10-30T20:45:59.300374+0000","last_clean":"2023-10-30T20:45:59.300374+0000","last_became_active":"2023-10-30T20:25:23.607599+0000","last_became_peered":"2023-10-30T20:25:23.607599+0000","last_unstale":"2023-10-30T20:45:59.300374+0000","last_undegraded":"2023-10-30T20:45:59.300374+0000","last_fullsized":"2023-10-30T20:45:59.300374+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:45:59.298300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:45:59.298300+0000","last_clean_scrub_stamp":"2023-10-30T20:45:59.298300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:10:33.842120+0000","scrub_duration":0.034934029999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":880,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:04.855387+0000","last_change":"2023-10-30T20:46:04.855387+0000","last_active":"2023-10-30T20:46:04.855387+0000","last_peered":"2023-10-30T20:46:04.855387+0000","last_clean":"2023-10-30T20:46:04.855387+0000","last_became_active":"2023-10-30T20:25:28.326296+0000","last_became_peered":"2023-10-30T20:25:28.326296+0000","last_unstale":"2023-10-30T20:46:04.855387+0000","last_undegraded":"2023-10-30T20:46:04.855387+0000","last_fullsized":"2023-10-30T20:46:04.855387+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:04.853002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:04.853002+0000","last_clean_scrub_stamp":"2023-10-30T20:46:04.853002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:52:09.667086+0000","scrub_duration":0.042164409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":867,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:04.130324+0000","last_change":"2023-10-30T20:46:04.130324+0000","last_active":"2023-10-30T20:46:04.130324+0000","last_peered":"2023-10-30T20:46:04.130324+0000","last_clean":"2023-10-30T20:46:04.130324+0000","last_became_active":"2023-10-30T20:25:20.671965+0000","last_became_peered":"2023-10-30T20:25:20.671965+0000","last_unstale":"2023-10-30T20:46:04.130324+0000","last_undegraded":"2023-10-30T20:46:04.130324+0000","last_fullsized":"2023-10-30T20:46:04.130324+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:04.124872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:04.124872+0000","last_clean_scrub_stamp":"2023-10-30T20:46:04.124872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T06:23:46.948945+0000","scrub_duration":0.038255145999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":880,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:13.880673+0000","last_change":"2023-10-30T20:46:13.880673+0000","last_active":"2023-10-30T20:46:13.880673+0000","last_peered":"2023-10-30T20:46:13.880673+0000","last_clean":"2023-10-30T20:46:13.880673+0000","last_became_active":"2023-10-30T20:25:28.979668+0000","last_became_peered":"2023-10-30T20:25:28.979668+0000","last_unstale":"2023-10-30T20:46:13.880673+0000","last_undegraded":"2023-10-30T20:46:13.880673+0000","last_fullsized":"2023-10-30T20:46:13.880673+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:13.878449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:13.878449+0000","last_clean_scrub_stamp":"2023-10-30T20:46:13.878449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:11:18.690935+0000","scrub_duration":0.038533900000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":880,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:08.954913+0000","last_change":"2023-10-30T20:46:08.954913+0000","last_active":"2023-10-30T20:46:08.954913+0000","last_peered":"2023-10-30T20:46:08.954913+0000","last_clean":"2023-10-30T20:46:08.954913+0000","last_became_active":"2023-10-30T20:25:23.058176+0000","last_became_peered":"2023-10-30T20:25:23.058176+0000","last_unstale":"2023-10-30T20:46:08.954913+0000","last_undegraded":"2023-10-30T20:46:08.954913+0000","last_fullsized":"2023-10-30T20:46:08.954913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:08.949899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:08.949899+0000","last_clean_scrub_stamp":"2023-10-30T20:46:08.949899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T06:23:50.453541+0000","scrub_duration":0.028102695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":881,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:09.652925+0000","last_change":"2023-10-30T20:46:09.653991+0000","last_active":"2023-10-30T20:46:09.652925+0000","last_peered":"2023-10-30T20:46:09.652925+0000","last_clean":"2023-10-30T20:46:09.652925+0000","last_became_active":"2023-10-30T20:25:28.798013+0000","last_became_peered":"2023-10-30T20:25:28.798013+0000","last_unstale":"2023-10-30T20:46:09.652925+0000","last_undegraded":"2023-10-30T20:46:09.652925+0000","last_fullsized":"2023-10-30T20:46:09.652925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:09.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:09.651479+0000","last_clean_scrub_stamp":"2023-10-30T20:46:09.651479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T07:34:16.105390+0000","scrub_duration":0.052863632000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":881,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:10.624227+0000","last_change":"2023-10-30T20:46:10.625301+0000","last_active":"2023-10-30T20:46:10.624227+0000","last_peered":"2023-10-30T20:46:10.624227+0000","last_clean":"2023-10-30T20:46:10.624227+0000","last_became_active":"2023-10-30T20:25:23.709112+0000","last_became_peered":"2023-10-30T20:25:23.709112+0000","last_unstale":"2023-10-30T20:46:10.624227+0000","last_undegraded":"2023-10-30T20:46:10.624227+0000","last_fullsized":"2023-10-30T20:46:10.624227+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:10.622841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:10.622841+0000","last_clean_scrub_stamp":"2023-10-30T20:46:10.622841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:21:16.161770+0000","scrub_duration":0.037042064999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":861,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:05.279023+0000","last_change":"2023-10-30T20:46:05.280404+0000","last_active":"2023-10-30T20:46:05.279023+0000","last_peered":"2023-10-30T20:46:05.279023+0000","last_clean":"2023-10-30T20:46:05.279023+0000","last_became_active":"2023-10-30T20:25:28.762953+0000","last_became_peered":"2023-10-30T20:25:28.762953+0000","last_unstale":"2023-10-30T20:46:05.279023+0000","last_undegraded":"2023-10-30T20:46:05.279023+0000","last_fullsized":"2023-10-30T20:46:05.279023+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:05.277120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:05.277120+0000","last_clean_scrub_stamp":"2023-10-30T20:46:05.277120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:09:47.711405+0000","scrub_duration":0.050602715999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":878,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:00.883410+0000","last_change":"2023-10-30T20:46:00.884420+0000","last_active":"2023-10-30T20:46:00.883410+0000","last_peered":"2023-10-30T20:46:00.883410+0000","last_clean":"2023-10-30T20:46:00.883410+0000","last_became_active":"2023-10-30T20:25:21.911569+0000","last_became_peered":"2023-10-30T20:25:21.911569+0000","last_unstale":"2023-10-30T20:46:00.883410+0000","last_undegraded":"2023-10-30T20:46:00.883410+0000","last_fullsized":"2023-10-30T20:46:00.883410+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:00.882062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:00.882062+0000","last_clean_scrub_stamp":"2023-10-30T20:46:00.882062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:34:26.631811+0000","scrub_duration":0.032802308000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":885,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:06.934699+0000","last_change":"2023-10-30T20:46:06.935680+0000","last_active":"2023-10-30T20:46:06.934699+0000","last_peered":"2023-10-30T20:46:06.934699+0000","last_clean":"2023-10-30T20:46:06.934699+0000","last_became_active":"2023-10-30T20:29:14.860294+0000","last_became_peered":"2023-10-30T20:29:14.860294+0000","last_unstale":"2023-10-30T20:46:06.934699+0000","last_undegraded":"2023-10-30T20:46:06.934699+0000","last_fullsized":"2023-10-30T20:46:06.934699+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:06.933427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:06.933427+0000","last_clean_scrub_stamp":"2023-10-30T20:46:06.933427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:03:24.607098+0000","scrub_duration":0.059309705999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":889,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-10-30T20:46:04.771463+0000","last_change":"2023-10-30T20:46:04.772433+0000","last_active":"2023-10-30T20:46:04.771463+0000","last_peered":"2023-10-30T20:46:04.771463+0000","last_clean":"2023-10-30T20:46:04.771463+0000","last_became_active":"2023-10-30T20:25:29.004167+0000","last_became_peered":"2023-10-30T20:25:29.004167+0000","last_unstale":"2023-10-30T20:46:04.771463+0000","last_undegraded":"2023-10-30T20:46:04.771463+0000","last_fullsized":"2023-10-30T20:46:04.771463+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T20:46:04.770139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T20:46:04.770139+0000","last_clean_scrub_stamp":"2023-10-30T20:46:04.770139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:54:39.727192+0000","scrub_duration":0.035219334999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477011,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59864,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311976,"statfs":{"total":96636764160,"available":96575463424,"internally_reserved":0,"allocated":6635520,"data_stored":3147947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:46:31 2023","interfaces":[{"interface":"back","average":{"1min":22.901,"5min":19.132000000000001,"15min":15.926},"min":{"1min":4.3499999999999996,"5min":4.3499999999999996,"15min":3.9740000000000002},"max":{"1min":229.53899999999999,"5min":229.53899999999999,"15min":229.53899999999999},"last":14.941000000000001},{"interface":"front","average":{"1min":27.510000000000002,"5min":21.686,"15min":17.997},"min":{"1min":6.109,"5min":4.9279999999999999,"15min":4.2539999999999996},"max":{"1min":194.93199999999999,"5min":219.797,"15min":219.797},"last":4.6609999999999996}]},{"osd":1,"last update":"Mon Oct 30 20:46:31 2023","interfaces":[{"interface":"back","average":{"1min":29.309999999999999,"5min":22.055,"15min":19.311},"min":{"1min":5.2309999999999999,"5min":5.2309999999999999,"15min":4.5940000000000003},"max":{"1min":232.976,"5min":232.976,"15min":232.976},"last":15.797000000000001},{"interface":"front","average":{"1min":23.454000000000001,"5min":20.331,"15min":19.966000000000001},"min":{"1min":6.5030000000000001,"5min":5.7320000000000002,"15min":5.726},"max":{"1min":211.82599999999999,"5min":237.88300000000001,"15min":237.88300000000001},"last":8.2469999999999999}]},{"osd":2,"last update":"Mon Oct 30 20:46:31 2023","interfaces":[{"interface":"back","average":{"1min":30.016999999999999,"5min":26.588999999999999,"15min":24.704999999999998},"min":{"1min":8.2739999999999991,"5min":7.0419999999999998,"15min":6.1539999999999999},"max":{"1min":236.99299999999999,"5min":267.81099999999998,"15min":835.96799999999996},"last":16.634},{"interface":"front","average":{"1min":32.875,"5min":28.224,"15min":23.370999999999999},"min":{"1min":11.957000000000001,"5min":10.252000000000001,"15min":8.2479999999999993},"max":{"1min":264.58300000000003,"5min":264.58300000000003,"15min":264.58300000000003},"last":17.434000000000001}]},{"osd":4,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":29.966000000000001,"5min":27.501000000000001,"15min":23.704999999999998},"min":{"1min":6.8390000000000004,"5min":6.625,"15min":5.5190000000000001},"max":{"1min":196.904,"5min":247.904,"15min":247.904},"last":13.834},{"interface":"front","average":{"1min":29.170999999999999,"5min":25.465,"15min":24.071000000000002},"min":{"1min":7.9900000000000002,"5min":6.8620000000000001,"15min":6.3840000000000003},"max":{"1min":210.88200000000001,"5min":235.64099999999999,"15min":235.64099999999999},"last":8.9920000000000009}]},{"osd":5,"last update":"Mon Oct 30 20:46:31 2023","interfaces":[{"interface":"back","average":{"1min":28.227,"5min":26.823,"15min":24.234000000000002},"min":{"1min":7.5419999999999998,"5min":6.726,"15min":6.726},"max":{"1min":226.994,"5min":233.28999999999999,"15min":233.28999999999999},"last":9.6310000000000002},{"interface":"front","average":{"1min":31.658999999999999,"5min":32.179000000000002,"15min":24.725999999999999},"min":{"1min":12.914,"5min":10.645,"15min":7.883},"max":{"1min":247.48400000000001,"5min":247.48400000000001,"15min":247.48400000000001},"last":18.367000000000001}]},{"osd":6,"last update":"Mon Oct 30 20:46:31 2023","interfaces":[{"interface":"back","average":{"1min":43.448,"5min":33.993000000000002,"15min":27.274000000000001},"min":{"1min":14.256,"5min":9.1989999999999998,"15min":9.1989999999999998},"max":{"1min":287.25700000000001,"5min":287.25700000000001,"15min":287.25700000000001},"last":18.954000000000001},{"interface":"front","average":{"1min":27.693000000000001,"5min":24.651,"15min":21.449000000000002},"min":{"1min":7.8120000000000003,"5min":6.093,"15min":6.093},"max":{"1min":209.03700000000001,"5min":262.53500000000003,"15min":262.53500000000003},"last":7.4660000000000002}]},{"osd":7,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":29.765999999999998,"5min":27.036000000000001,"15min":24.699000000000002},"min":{"1min":9.7669999999999995,"5min":8.5570000000000004,"15min":7.3799999999999999},"max":{"1min":222.32400000000001,"5min":264.48000000000002,"15min":264.48000000000002},"last":13.16},{"interface":"front","average":{"1min":31.981999999999999,"5min":27.591000000000001,"15min":26.734999999999999},"min":{"1min":9.2919999999999998,"5min":8.9700000000000006,"15min":8.9700000000000006},"max":{"1min":233.99700000000001,"5min":245.042,"15min":245.042},"last":10.353999999999999}]}]},{"osd":1,"up_from":16,"seq":68719477011,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64060,"kb_used_data":6516,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94307780,"statfs":{"total":96636764160,"available":96571166720,"internally_reserved":0,"allocated":6672384,"data_stored":3545787,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.762,"5min":15.228999999999999,"15min":16.553999999999998},"min":{"1min":4.1130000000000004,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":98.356999999999999,"5min":244.65000000000001,"15min":244.65000000000001},"last":4.4429999999999996},{"interface":"front","average":{"1min":11.536,"5min":12.617000000000001,"15min":18.32},"min":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":3.847},"max":{"1min":39.68,"5min":65.853999999999999,"15min":407.38900000000001},"last":6.8650000000000002}]},{"osd":2,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":18.402000000000001,"5min":16.823,"15min":19.931000000000001},"min":{"1min":7.5389999999999997,"5min":7.367,"15min":4.1500000000000004},"max":{"1min":80.688000000000002,"5min":113.649,"15min":297.26499999999999},"last":10.611000000000001},{"interface":"front","average":{"1min":23.346,"5min":19.442,"15min":20.483000000000001},"min":{"1min":9.8040000000000003,"5min":8.5690000000000008,"15min":6.5469999999999997},"max":{"1min":148.03999999999999,"5min":148.03999999999999,"15min":167.12100000000001},"last":11.711}]},{"osd":3,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":17.946000000000002,"5min":19.791,"15min":21.096},"min":{"1min":8.8569999999999993,"5min":7.8029999999999999,"15min":6.5990000000000002},"max":{"1min":84.090999999999994,"5min":150.166,"15min":213.971},"last":13.848000000000001},{"interface":"front","average":{"1min":19.449000000000002,"5min":22.609000000000002,"15min":24.210999999999999},"min":{"1min":12.093999999999999,"5min":9.8409999999999993,"15min":8.1449999999999996},"max":{"1min":74.275999999999996,"5min":146.15899999999999,"15min":223.49199999999999},"last":16.091000000000001}]},{"osd":4,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":13.090999999999999,"5min":20.369,"15min":21.076000000000001},"min":{"1min":8.1349999999999998,"5min":6.8659999999999997,"15min":5.9809999999999999},"max":{"1min":30.533000000000001,"5min":200.98400000000001,"15min":215.435},"last":12.336},{"interface":"front","average":{"1min":17.289000000000001,"5min":23.798999999999999,"15min":25.364000000000001},"min":{"1min":12.874000000000001,"5min":11.516999999999999,"15min":8.9939999999999998},"max":{"1min":26.600000000000001,"5min":192.69200000000001,"15min":197.34},"last":14.475}]},{"osd":5,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":13.349,"5min":19.747,"15min":19.356000000000002},"min":{"1min":7.4020000000000001,"5min":6.4580000000000002,"15min":5.1390000000000002},"max":{"1min":62.798000000000002,"5min":209.35499999999999,"15min":241.798},"last":8.016},{"interface":"front","average":{"1min":17.783000000000001,"5min":20.292000000000002,"15min":21.498000000000001},"min":{"1min":9.5739999999999998,"5min":6.6230000000000002,"15min":6.6230000000000002},"max":{"1min":68.882000000000005,"5min":118.65600000000001,"15min":248.124},"last":14.779}]},{"osd":6,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.948,"5min":20.902000000000001,"15min":23.236999999999998},"min":{"1min":10.205,"5min":7.7220000000000004,"15min":6.9880000000000004},"max":{"1min":37.037999999999997,"5min":204.71899999999999,"15min":406.72399999999999},"last":16.84},{"interface":"front","average":{"1min":28.690999999999999,"5min":25.748000000000001,"15min":25.416},"min":{"1min":11.704000000000001,"5min":10.397,"15min":8.4749999999999996},"max":{"1min":179.98099999999999,"5min":179.98099999999999,"15min":179.98099999999999},"last":18.684999999999999}]},{"osd":7,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":14.643000000000001,"5min":18.478999999999999,"15min":23.765000000000001},"min":{"1min":6.0410000000000004,"5min":6.0410000000000004,"15min":6.0110000000000001},"max":{"1min":58.841999999999999,"5min":188.792,"15min":249.88300000000001},"last":8.7100000000000009},{"interface":"front","average":{"1min":19.361000000000001,"5min":21.233000000000001,"15min":25.628},"min":{"1min":12.742000000000001,"5min":8.8559999999999999,"15min":7.1360000000000001},"max":{"1min":59.718000000000004,"5min":134.89400000000001,"15min":275.36700000000002},"last":17.495000000000001}]}]},{"osd":0,"up_from":16,"seq":68719477012,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59868,"kb_used_data":6484,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311972,"statfs":{"total":96636764160,"available":96575459328,"internally_reserved":0,"allocated":6639616,"data_stored":3147952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 20:46:40 2023","interfaces":[{"interface":"back","average":{"1min":10.099,"5min":11.699,"15min":17.577999999999999},"min":{"1min":4.7480000000000002,"5min":4.1740000000000004,"15min":4.0819999999999999},"max":{"1min":16.696000000000002,"5min":64.745999999999995,"15min":341.85300000000001},"last":11.002000000000001},{"interface":"front","average":{"1min":9.9529999999999994,"5min":14.31,"15min":17.991},"min":{"1min":5.734,"5min":5.734,"15min":5.1669999999999998},"max":{"1min":21.934999999999999,"5min":77.805999999999997,"15min":243.62100000000001},"last":5.734}]},{"osd":2,"last update":"Mon Oct 30 20:46:01 2023","interfaces":[{"interface":"back","average":{"1min":18.695,"5min":19.710999999999999,"15min":25.018000000000001},"min":{"1min":6.0309999999999997,"5min":5.298,"15min":4.6449999999999996},"max":{"1min":93.063000000000002,"5min":199.71100000000001,"15min":580.43100000000004},"last":13.025},{"interface":"front","average":{"1min":31.413,"5min":21.189,"15min":26.045000000000002},"min":{"1min":6.0910000000000002,"5min":6.0910000000000002,"15min":6.0910000000000002},"max":{"1min":292.03699999999998,"5min":292.03699999999998,"15min":1090.171},"last":9.2040000000000006}]},{"osd":3,"last update":"Mon Oct 30 20:46:40 2023","interfaces":[{"interface":"back","average":{"1min":9.1080000000000005,"5min":12.914999999999999,"15min":16.361999999999998},"min":{"1min":5.7789999999999999,"5min":5.7789999999999999,"15min":5.298},"max":{"1min":15.061,"5min":90.641000000000005,"15min":167.607},"last":11.388},{"interface":"front","average":{"1min":13.643000000000001,"5min":21.097000000000001,"15min":24.164999999999999},"min":{"1min":6.8710000000000004,"5min":6.8710000000000004,"15min":6.4370000000000003},"max":{"1min":23.213999999999999,"5min":290.19600000000003,"15min":318.245},"last":6.8710000000000004}]},{"osd":4,"last update":"Mon Oct 30 20:46:40 2023","interfaces":[{"interface":"back","average":{"1min":11.872,"5min":18.486000000000001,"15min":19.120999999999999},"min":{"1min":7.8600000000000003,"5min":7.1059999999999999,"15min":5.8170000000000002},"max":{"1min":17.055,"5min":116.98399999999999,"15min":174.67699999999999},"last":13.651999999999999},{"interface":"front","average":{"1min":15.525,"5min":24.631,"15min":26.198},"min":{"1min":7.9290000000000003,"5min":7.9290000000000003,"15min":7.5830000000000002},"max":{"1min":33.905999999999999,"5min":153.124,"15min":242.06800000000001},"last":7.9290000000000003}]},{"osd":5,"last update":"Mon Oct 30 20:46:40 2023","interfaces":[{"interface":"back","average":{"1min":13.409000000000001,"5min":21.062000000000001,"15min":21.57},"min":{"1min":7.4429999999999996,"5min":7.4429999999999996,"15min":6.8099999999999996},"max":{"1min":26.283999999999999,"5min":170.024,"15min":170.024},"last":12.401999999999999},{"interface":"front","average":{"1min":14.805999999999999,"5min":21.643999999999998,"15min":21.713000000000001},"min":{"1min":8.6329999999999991,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":25.364999999999998,"5min":177.41800000000001,"15min":177.41800000000001},"last":14.282999999999999}]},{"osd":6,"last update":"Mon Oct 30 20:46:40 2023","interfaces":[{"interface":"back","average":{"1min":17.413,"5min":26.440999999999999,"15min":29.007000000000001},"min":{"1min":8.5229999999999997,"5min":8.5229999999999997,"15min":8.2590000000000003},"max":{"1min":34.863999999999997,"5min":178.756,"15min":330.37799999999999},"last":8.5229999999999997},{"interface":"front","average":{"1min":16.309999999999999,"5min":19.050999999999998,"15min":23.341000000000001},"min":{"1min":8.6129999999999995,"5min":6.8410000000000002,"15min":6.8410000000000002},"max":{"1min":28.603999999999999,"5min":77.477000000000004,"15min":201.50299999999999},"last":17.129000000000001}]},{"osd":7,"last update":"Mon Oct 30 20:46:40 2023","interfaces":[{"interface":"back","average":{"1min":16.920000000000002,"5min":21.317,"15min":26.120000000000001},"min":{"1min":11.134,"5min":10.81,"15min":9.6820000000000004},"max":{"1min":30.404,"5min":77.103999999999999,"15min":268.18900000000002},"last":14.935},{"interface":"front","average":{"1min":16.414999999999999,"5min":19.48,"15min":24.419},"min":{"1min":8.2409999999999997,"5min":8.2409999999999997,"15min":7.3559999999999999},"max":{"1min":27.532,"5min":77.927000000000007,"15min":228.495},"last":15.539}]}]},{"osd":6,"up_from":15,"seq":64424509716,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59864,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311976,"statfs":{"total":96636764160,"available":96575463424,"internally_reserved":0,"allocated":6635520,"data_stored":3147947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:49 2023","interfaces":[{"interface":"back","average":{"1min":15.789,"5min":12.747,"15min":13.856999999999999},"min":{"1min":4.3819999999999997,"5min":4.0179999999999998,"15min":3.7080000000000002},"max":{"1min":66.393000000000001,"5min":73.316000000000003,"15min":151.00299999999999},"last":5.125},{"interface":"front","average":{"1min":10.926,"5min":11.757,"15min":15.211},"min":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"max":{"1min":53.823999999999998,"5min":75.060000000000002,"15min":146.49299999999999},"last":9.2170000000000005}]},{"osd":1,"last update":"Mon Oct 30 20:45:49 2023","interfaces":[{"interface":"back","average":{"1min":14.625,"5min":16.163,"15min":19.315999999999999},"min":{"1min":5.8049999999999997,"5min":5.4379999999999997,"15min":4.9199999999999999},"max":{"1min":74.001000000000005,"5min":117.193,"15min":218.78100000000001},"last":10.436},{"interface":"front","average":{"1min":10.275,"5min":13.943,"15min":15.449},"min":{"1min":5.7229999999999999,"5min":5.1379999999999999,"15min":5.0149999999999997},"max":{"1min":27.335999999999999,"5min":128.50200000000001,"15min":155.27199999999999},"last":6.6900000000000004}]},{"osd":2,"last update":"Mon Oct 30 20:46:34 2023","interfaces":[{"interface":"back","average":{"1min":12.077,"5min":14.657999999999999,"15min":19.649999999999999},"min":{"1min":7.8840000000000003,"5min":6.4459999999999997,"15min":6.1870000000000003},"max":{"1min":18.312999999999999,"5min":67.552999999999997,"15min":222.303},"last":13.268000000000001},{"interface":"front","average":{"1min":12.42,"5min":16.469999999999999,"15min":24.222000000000001},"min":{"1min":8.1869999999999994,"5min":7.9580000000000002,"15min":6.1820000000000004},"max":{"1min":17.215,"5min":98.781000000000006,"15min":438.19200000000001},"last":12.669}]},{"osd":3,"last update":"Mon Oct 30 20:46:39 2023","interfaces":[{"interface":"back","average":{"1min":15.201000000000001,"5min":15.677,"15min":17.09},"min":{"1min":6.2919999999999998,"5min":6.2050000000000001,"15min":6.0179999999999998},"max":{"1min":105.062,"5min":136.16800000000001,"15min":217.99000000000001},"last":8.125},{"interface":"front","average":{"1min":20.588000000000001,"5min":20.905999999999999,"15min":20.916},"min":{"1min":6.6340000000000003,"5min":6.3890000000000002,"15min":6.0010000000000003},"max":{"1min":153.32900000000001,"5min":270.25599999999997,"15min":270.25599999999997},"last":14.276999999999999}]},{"osd":4,"last update":"Mon Oct 30 20:46:39 2023","interfaces":[{"interface":"back","average":{"1min":16.032,"5min":19.484000000000002,"15min":21.670999999999999},"min":{"1min":10.561999999999999,"5min":8.1539999999999999,"15min":7.6429999999999998},"max":{"1min":34.412999999999997,"5min":168.38800000000001,"15min":190.29599999999999},"last":15.085000000000001},{"interface":"front","average":{"1min":13.824,"5min":16.885999999999999,"15min":20.181999999999999},"min":{"1min":8.7420000000000009,"5min":6.5199999999999996,"15min":6.5199999999999996},"max":{"1min":18.524000000000001,"5min":112.042,"15min":189.59399999999999},"last":17.648}]},{"osd":5,"last update":"Mon Oct 30 20:46:34 2023","interfaces":[{"interface":"back","average":{"1min":16.443999999999999,"5min":18.797000000000001,"15min":20.283000000000001},"min":{"1min":11.276999999999999,"5min":8.9939999999999998,"15min":8.9930000000000003},"max":{"1min":24.062000000000001,"5min":68.114000000000004,"15min":138.38800000000001},"last":16.399000000000001},{"interface":"front","average":{"1min":18.379000000000001,"5min":21.609000000000002,"15min":23.030999999999999},"min":{"1min":13.930999999999999,"5min":11.9,"15min":10.571},"max":{"1min":28.832000000000001,"5min":68.138999999999996,"15min":150.35499999999999},"last":18.332999999999998}]},{"osd":7,"last update":"Mon Oct 30 20:46:34 2023","interfaces":[{"interface":"back","average":{"1min":13.680999999999999,"5min":15.797000000000001,"15min":21.291},"min":{"1min":7.9420000000000002,"5min":7.1699999999999999,"15min":6.226},"max":{"1min":39.084000000000003,"5min":77.778000000000006,"15min":253.16900000000001},"last":11.504},{"interface":"front","average":{"1min":14.586,"5min":18.219000000000001,"15min":18.826000000000001},"min":{"1min":11.497,"5min":7.4029999999999996,"15min":5.7309999999999999},"max":{"1min":19.113,"5min":75.846000000000004,"15min":221.28399999999999},"last":15.725}]}]},{"osd":5,"up_from":15,"seq":64424509713,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77948,"kb_used_data":6516,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293892,"statfs":{"total":96636764160,"available":96556945408,"internally_reserved":0,"allocated":6672384,"data_stored":3545787,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":17.625,"5min":18.785,"15min":16.260999999999999},"min":{"1min":5.2329999999999997,"5min":3.8359999999999999,"15min":3.8359999999999999},"max":{"1min":85.513999999999996,"5min":220.148,"15min":220.148},"last":13.269},{"interface":"front","average":{"1min":27.861000000000001,"5min":23.675000000000001,"15min":20.949000000000002},"min":{"1min":8.6039999999999992,"5min":5.6059999999999999,"15min":5.6059999999999999},"max":{"1min":145.43000000000001,"5min":177.91,"15min":177.91},"last":14.590999999999999}]},{"osd":1,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":20.715,"5min":16.585000000000001,"15min":20.353999999999999},"min":{"1min":6.3040000000000003,"5min":4.3010000000000002,"15min":4.3010000000000002},"max":{"1min":71.460999999999999,"5min":74.081000000000003,"15min":293.75400000000002},"last":6.8929999999999998},{"interface":"front","average":{"1min":20.943999999999999,"5min":16.175000000000001,"15min":18.327999999999999},"min":{"1min":5.46,"5min":4.8949999999999996,"15min":4.8949999999999996},"max":{"1min":80.397000000000006,"5min":80.397000000000006,"15min":210.797},"last":8.3339999999999996}]},{"osd":2,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":31.280000000000001,"5min":23.431999999999999,"15min":25.231000000000002},"min":{"1min":6.2480000000000002,"5min":5.0209999999999999,"15min":5.0209999999999999},"max":{"1min":105.449,"5min":143.273,"15min":519.68799999999999},"last":15.904},{"interface":"front","average":{"1min":23.294,"5min":20.007000000000001,"15min":22.029},"min":{"1min":5.6479999999999997,"5min":5.6479999999999997,"15min":5.226},"max":{"1min":82.042000000000002,"5min":154.083,"15min":218.59899999999999},"last":7.7619999999999996}]},{"osd":3,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":41.817,"5min":31.161999999999999,"15min":26.263000000000002},"min":{"1min":9.9900000000000002,"5min":9.9900000000000002,"15min":7.6150000000000002},"max":{"1min":269.52999999999997,"5min":269.52999999999997,"15min":269.52999999999997},"last":15.361000000000001},{"interface":"front","average":{"1min":24.488,"5min":22.629000000000001,"15min":22.704000000000001},"min":{"1min":7.4480000000000004,"5min":6.218,"15min":5.9119999999999999},"max":{"1min":178.78700000000001,"5min":178.78700000000001,"15min":249.797},"last":9.0739999999999998}]},{"osd":4,"last update":"Mon Oct 30 20:45:47 2023","interfaces":[{"interface":"back","average":{"1min":21.725999999999999,"5min":26.437999999999999,"15min":25.25},"min":{"1min":8.1080000000000005,"5min":7.1580000000000004,"15min":6.6100000000000003},"max":{"1min":78.555999999999997,"5min":215.97999999999999,"15min":291.92599999999999},"last":10.576000000000001},{"interface":"front","average":{"1min":26.097000000000001,"5min":33.600000000000001,"15min":28.585000000000001},"min":{"1min":11.895,"5min":8.0050000000000008,"15min":6.9550000000000001},"max":{"1min":70.290999999999997,"5min":324.31,"15min":324.31},"last":11.843999999999999}]},{"osd":6,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":32.883000000000003,"5min":28.116,"15min":26.379999999999999},"min":{"1min":10.593,"5min":8.6489999999999991,"15min":7.2199999999999998},"max":{"1min":162.964,"5min":211.255,"15min":251.37700000000001},"last":11.321},{"interface":"front","average":{"1min":38.536000000000001,"5min":33.917999999999999,"15min":28.524000000000001},"min":{"1min":9.3640000000000008,"5min":9.2349999999999994,"15min":7.1619999999999999},"max":{"1min":160.42599999999999,"5min":333.29700000000003,"15min":333.29700000000003},"last":16.759}]},{"osd":7,"last update":"Mon Oct 30 20:45:38 2023","interfaces":[{"interface":"back","average":{"1min":36.252000000000002,"5min":33.210000000000001,"15min":27.071999999999999},"min":{"1min":11.131,"5min":10.68,"15min":6.1769999999999996},"max":{"1min":116.877,"5min":258.81,"15min":264.52999999999997},"last":17.516999999999999},{"interface":"front","average":{"1min":26.748000000000001,"5min":22.239000000000001,"15min":25.803999999999998},"min":{"1min":7.4580000000000002,"5min":5.8410000000000002,"15min":5.8410000000000002},"max":{"1min":90.570999999999998,"5min":154.86799999999999,"15min":255.625},"last":12.598000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542418,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59864,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311976,"statfs":{"total":96636764160,"available":96575463424,"internally_reserved":0,"allocated":6635520,"data_stored":3147947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":9.2010000000000005,"5min":13.369999999999999,"15min":14.285},"min":{"1min":4.9580000000000002,"5min":4.0499999999999998,"15min":3.5920000000000001},"max":{"1min":18.266999999999999,"5min":106.79300000000001,"15min":187.85499999999999},"last":8.1920000000000002},{"interface":"front","average":{"1min":8.7729999999999997,"5min":12.766999999999999,"15min":13.725},"min":{"1min":5.0279999999999996,"5min":4.1020000000000003,"15min":4.1020000000000003},"max":{"1min":15.933,"5min":117.645,"15min":124.342},"last":5.7400000000000002}]},{"osd":1,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":13.706,"5min":18.475999999999999,"15min":19.167999999999999},"min":{"1min":8.1969999999999992,"5min":5.8979999999999997,"15min":5.7409999999999997},"max":{"1min":33.970999999999997,"5min":103.907,"15min":226.63399999999999},"last":9.1489999999999991},{"interface":"front","average":{"1min":11.929,"5min":17.565999999999999,"15min":19.605},"min":{"1min":6.5439999999999996,"5min":6.351,"15min":5.859},"max":{"1min":42.719000000000001,"5min":135.41200000000001,"15min":187.24100000000001},"last":14.137}]},{"osd":2,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":17.675000000000001,"5min":18.248999999999999,"15min":20.965},"min":{"1min":7.2869999999999999,"5min":6.2549999999999999,"15min":6.2549999999999999},"max":{"1min":43.064999999999998,"5min":79.168000000000006,"15min":200.285},"last":15.558},{"interface":"front","average":{"1min":26.209,"5min":21.07,"15min":21.149999999999999},"min":{"1min":6.0679999999999996,"5min":5.5090000000000003,"15min":5.5090000000000003},"max":{"1min":211.315,"5min":211.315,"15min":241.15700000000001},"last":7.5549999999999997}]},{"osd":3,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":25.942,"5min":21.164000000000001,"15min":22.437999999999999},"min":{"1min":10.516,"5min":8.1509999999999998,"15min":6.6849999999999996},"max":{"1min":205.821,"5min":205.821,"15min":262.43900000000002},"last":14.802},{"interface":"front","average":{"1min":26.780000000000001,"5min":22.042999999999999,"15min":23.443999999999999},"min":{"1min":7.5650000000000004,"5min":6.7460000000000004,"15min":5.9950000000000001},"max":{"1min":225.29900000000001,"5min":225.29900000000001,"15min":330.15300000000002},"last":12.369}]},{"osd":4,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.153,"5min":21.227,"15min":22.390000000000001},"min":{"1min":9.9459999999999997,"5min":7.218,"15min":7.218},"max":{"1min":22.241,"5min":136.494,"15min":213.072},"last":16.783999999999999},{"interface":"front","average":{"1min":17.73,"5min":22.32,"15min":24.960999999999999},"min":{"1min":8.9540000000000006,"5min":8.9540000000000006,"15min":8.5530000000000008},"max":{"1min":28.087,"5min":104.538,"15min":239.76900000000001},"last":17.606999999999999}]},{"osd":5,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":15.374000000000001,"5min":18.962,"15min":22.965},"min":{"1min":10.206,"5min":8.2889999999999997,"15min":7.258},"max":{"1min":27.122,"5min":110.45099999999999,"15min":275.08100000000002},"last":11.253},{"interface":"front","average":{"1min":13.379,"5min":15.702999999999999,"15min":18.018000000000001},"min":{"1min":5.3410000000000002,"5min":5.3410000000000002,"15min":5.3410000000000002},"max":{"1min":40.390000000000001,"5min":94.302999999999997,"15min":129.935},"last":10.509}]},{"osd":6,"last update":"Mon Oct 30 20:45:39 2023","interfaces":[{"interface":"back","average":{"1min":18.283000000000001,"5min":21.891999999999999,"15min":25.209},"min":{"1min":11.725,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":27.867999999999999,"5min":126.001,"15min":263.41399999999999},"last":18.411000000000001},{"interface":"front","average":{"1min":17.962,"5min":19.896000000000001,"15min":23.355},"min":{"1min":11.305999999999999,"5min":8.0229999999999997,"15min":8.0229999999999997},"max":{"1min":56.701000000000001,"5min":111.691,"15min":257.75900000000001},"last":12.975}]}]},{"osd":2,"up_from":14,"seq":60129542418,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82076,"kb_used_data":6484,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289764,"statfs":{"total":96636764160,"available":96552718336,"internally_reserved":0,"allocated":6639616,"data_stored":3147952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:46:32 2023","interfaces":[{"interface":"back","average":{"1min":24.350999999999999,"5min":16,"15min":18.062999999999999},"min":{"1min":5.4470000000000001,"5min":4.3170000000000002,"15min":4.3170000000000002},"max":{"1min":269.779,"5min":269.779,"15min":389.31400000000002},"last":7.5640000000000001},{"interface":"front","average":{"1min":14.538,"5min":17.489999999999998,"15min":18.309999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":4.4589999999999996},"max":{"1min":124.283,"5min":295.87,"15min":295.87},"last":7.8810000000000002}]},{"osd":1,"last update":"Mon Oct 30 20:45:48 2023","interfaces":[{"interface":"back","average":{"1min":14.757,"5min":16.274999999999999,"15min":22.295999999999999},"min":{"1min":6.4720000000000004,"5min":5.5060000000000002,"15min":5.2649999999999997},"max":{"1min":36.689,"5min":74.575000000000003,"15min":245.84999999999999},"last":13.657999999999999},{"interface":"front","average":{"1min":17.843,"5min":15.342000000000001,"15min":20.946999999999999},"min":{"1min":7.9459999999999997,"5min":5.8399999999999999,"15min":5.3330000000000002},"max":{"1min":66.947999999999993,"5min":72.662999999999997,"15min":317.20699999999999},"last":14.029}]},{"osd":3,"last update":"Mon Oct 30 20:45:40 2023","interfaces":[{"interface":"back","average":{"1min":16.094000000000001,"5min":17.843,"15min":22.158000000000001},"min":{"1min":6.6040000000000001,"5min":6.1459999999999999,"15min":5.2619999999999996},"max":{"1min":58.442999999999998,"5min":253.02799999999999,"15min":311.03100000000001},"last":9.8249999999999993},{"interface":"front","average":{"1min":20.954000000000001,"5min":21.210999999999999,"15min":28.140999999999998},"min":{"1min":10.403,"5min":7.4969999999999999,"15min":6.2039999999999997},"max":{"1min":86.935000000000002,"5min":145.066,"15min":703},"last":8.8979999999999997}]},{"osd":4,"last update":"Mon Oct 30 20:46:32 2023","interfaces":[{"interface":"back","average":{"1min":13.628,"5min":16.239999999999998,"15min":21.978999999999999},"min":{"1min":6.1189999999999998,"5min":6.1189999999999998,"15min":6.1189999999999998},"max":{"1min":42.828000000000003,"5min":123.06399999999999,"15min":709.85900000000004},"last":11.291},{"interface":"front","average":{"1min":15.756,"5min":17.905999999999999,"15min":20.34},"min":{"1min":9.2729999999999997,"5min":6.2359999999999998,"15min":6.2359999999999998},"max":{"1min":39.279000000000003,"5min":118.82599999999999,"15min":182.947},"last":15.103}]},{"osd":5,"last update":"Mon Oct 30 20:46:32 2023","interfaces":[{"interface":"back","average":{"1min":25.829000000000001,"5min":19.032,"15min":25.166},"min":{"1min":7.9729999999999999,"5min":7.2279999999999998,"15min":6.8460000000000001},"max":{"1min":216.15000000000001,"5min":216.15000000000001,"15min":874.22799999999995},"last":10.661},{"interface":"front","average":{"1min":21.870999999999999,"5min":18.494,"15min":21.928000000000001},"min":{"1min":10.569000000000001,"5min":6.9500000000000002,"15min":6.9500000000000002},"max":{"1min":88.405000000000001,"5min":88.405000000000001,"15min":222.76400000000001},"last":15.972}]},{"osd":6,"last update":"Mon Oct 30 20:46:32 2023","interfaces":[{"interface":"back","average":{"1min":16.866,"5min":21.265999999999998,"15min":23.167999999999999},"min":{"1min":8.7449999999999992,"5min":8.5389999999999997,"15min":7.3049999999999997},"max":{"1min":55.75,"5min":152.47499999999999,"15min":238.84800000000001},"last":16.614999999999998},{"interface":"front","average":{"1min":19.239000000000001,"5min":21.538,"15min":25.091999999999999},"min":{"1min":11.028,"5min":8.0670000000000002,"15min":8.0670000000000002},"max":{"1min":56.610999999999997,"5min":161.30799999999999,"15min":250.06700000000001},"last":17.859000000000002}]},{"osd":7,"last update":"Mon Oct 30 20:46:32 2023","interfaces":[{"interface":"back","average":{"1min":21.399000000000001,"5min":21.239000000000001,"15min":24.994},"min":{"1min":11.845000000000001,"5min":9.7300000000000004,"15min":9.7300000000000004},"max":{"1min":87.628,"5min":133.429,"15min":354.904},"last":18.698},{"interface":"front","average":{"1min":15.561,"5min":20.600000000000001,"15min":26.390000000000001},"min":{"1min":9.4830000000000005,"5min":7.3440000000000003,"15min":7.3440000000000003},"max":{"1min":39.965000000000003,"5min":153.803,"15min":865.25800000000004},"last":11.882999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575124,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82072,"kb_used_data":6480,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289768,"statfs":{"total":96636764160,"available":96552722432,"internally_reserved":0,"allocated":6635520,"data_stored":3147947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 20:46:37 2023","interfaces":[{"interface":"back","average":{"1min":10.326000000000001,"5min":9.9659999999999993,"15min":13.890000000000001},"min":{"1min":4.6280000000000001,"5min":3.8839999999999999,"15min":3.839},"max":{"1min":33.112000000000002,"5min":73.688000000000002,"15min":224.95099999999999},"last":5.0679999999999996},{"interface":"front","average":{"1min":10.602,"5min":11.175000000000001,"15min":14.334},"min":{"1min":5.1280000000000001,"5min":4.407,"15min":4.0229999999999997},"max":{"1min":41.756,"5min":70.055999999999997,"15min":227.71600000000001},"last":8.5459999999999994}]},{"osd":1,"last update":"Mon Oct 30 20:46:37 2023","interfaces":[{"interface":"back","average":{"1min":11.477,"5min":14.965,"15min":18.263000000000002},"min":{"1min":5.774,"5min":5.4119999999999999,"15min":4.7960000000000003},"max":{"1min":32.235999999999997,"5min":118.747,"15min":203.68199999999999},"last":7.6369999999999996},{"interface":"front","average":{"1min":15.871,"5min":18.965,"15min":21.815000000000001},"min":{"1min":9.8190000000000008,"5min":8.3000000000000007,"15min":7.5},"max":{"1min":34.045999999999999,"5min":110.129,"15min":228.34999999999999},"last":16.530000000000001}]},{"osd":2,"last update":"Mon Oct 30 20:46:37 2023","interfaces":[{"interface":"back","average":{"1min":13.327,"5min":14.247999999999999,"15min":19.306999999999999},"min":{"1min":7.2800000000000002,"5min":4.6420000000000003,"15min":4.6420000000000003},"max":{"1min":36.182000000000002,"5min":82.471999999999994,"15min":347.30799999999999},"last":7.9630000000000001},{"interface":"front","average":{"1min":13.83,"5min":16.393000000000001,"15min":19.873999999999999},"min":{"1min":7.9400000000000004,"5min":6.1369999999999996,"15min":6.0090000000000003},"max":{"1min":44.536000000000001,"5min":121.45999999999999,"15min":183.03100000000001},"last":9.5229999999999997}]},{"osd":3,"last update":"Mon Oct 30 20:46:37 2023","interfaces":[{"interface":"back","average":{"1min":16.036000000000001,"5min":18.015000000000001,"15min":22.271000000000001},"min":{"1min":8.0890000000000004,"5min":6.4779999999999998,"15min":6.2930000000000001},"max":{"1min":47.680999999999997,"5min":174.733,"15min":249.31399999999999},"last":13.042},{"interface":"front","average":{"1min":18.555,"5min":18.492999999999999,"15min":22.103000000000002},"min":{"1min":9.3510000000000009,"5min":6.0220000000000002,"15min":6.0220000000000002},"max":{"1min":54.853000000000002,"5min":177.87,"15min":242.40000000000001},"last":17.344999999999999}]},{"osd":5,"last update":"Mon Oct 30 20:46:37 2023","interfaces":[{"interface":"back","average":{"1min":18.518999999999998,"5min":20.756,"15min":23.088999999999999},"min":{"1min":12.069000000000001,"5min":6.5839999999999996,"15min":6.5839999999999996},"max":{"1min":43.244,"5min":158.541,"15min":253.80199999999999},"last":18.154},{"interface":"front","average":{"1min":13.861000000000001,"5min":17.099,"15min":19.449999999999999},"min":{"1min":6.9240000000000004,"5min":6.1029999999999998,"15min":6.1029999999999998},"max":{"1min":45.380000000000003,"5min":147.54400000000001,"15min":271.572},"last":12.010999999999999}]},{"osd":6,"last update":"Mon Oct 30 20:46:37 2023","interfaces":[{"interface":"back","average":{"1min":16.399000000000001,"5min":18.681000000000001,"15min":23.611000000000001},"min":{"1min":6.7050000000000001,"5min":6.4210000000000003,"15min":6.4210000000000003},"max":{"1min":47.064,"5min":245.447,"15min":254.44399999999999},"last":14.391},{"interface":"front","average":{"1min":19.366,"5min":19.856999999999999,"15min":27.591000000000001},"min":{"1min":12.68,"5min":9.0419999999999998,"15min":8.1639999999999997},"max":{"1min":43.889000000000003,"5min":74.463999999999999,"15min":291.62599999999998},"last":18.975999999999999}]},{"osd":7,"last update":"Mon Oct 30 20:46:37 2023","interfaces":[{"interface":"back","average":{"1min":16.673999999999999,"5min":15.974,"15min":21.343},"min":{"1min":10.087999999999999,"5min":8.1890000000000001,"15min":6.2389999999999999},"max":{"1min":48.820999999999998,"5min":49.243000000000002,"15min":262.82100000000003},"last":13.69},{"interface":"front","average":{"1min":18.870000000000001,"5min":19.565000000000001,"15min":24.719999999999999},"min":{"1min":9.7560000000000002,"5min":9.7560000000000002,"15min":9.7560000000000002},"max":{"1min":55.171999999999997,"5min":61.091000000000001,"15min":244.37799999999999},"last":15.571999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T20:46:43.058 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-30T20:46:43.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.154+0000 7f532f056640 10 monclient: get_monmap_and_config 2023-10-30T20:46:43.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.154+0000 7f532f056640 10 monclient: build_initial_monmap 2023-10-30T20:46:43.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.154+0000 7f532f056640 10 monclient: monmap: 2023-10-30T20:46:43.156 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:46:43.156 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:43.156 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:46:43.155417+0000 2023-10-30T20:46:43.156 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:46:43.155417+0000 2023-10-30T20:46:43.157 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:46:43.157 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:43.157 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-10-30T20:46:43.157 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c-legacy 2023-10-30T20:46:43.157 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-10-30T20:46:43.158 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-10-30T20:46:43.158 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.183:3300/0 mon.noname-c 2023-10-30T20:46:43.158 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:43.160 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 1 Processor -- start 2023-10-30T20:46:43.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 1 -- start start 2023-10-30T20:46:43.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 10 monclient: init 2023-10-30T20:46:43.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 10 monclient: _reopen_session rank -1 2023-10-30T20:46:43.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-10-30T20:46:43.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f532806ecf0 0x7f5328070860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:43.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 10 monclient(hunting): picked mon.noname-c con 0x7f532806ecf0 addr v2:172.21.15.183:3300/0 2023-10-30T20:46:43.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532806f0d0 0x7f532806e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:43.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 10 monclient(hunting): picked mon.noname-b con 0x7f532806f0d0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:46:43.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f532813ba30 addr v1:172.21.15.69:6789/0 2023-10-30T20:46:43.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5328072340 con 0x7f532813ba30 2023-10-30T20:46:43.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:43.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5328067a30 con 0x7f532806ecf0 2023-10-30T20:46:43.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:43.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5328070da0 con 0x7f532806f0d0 2023-10-30T20:46:43.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 10 monclient(hunting): _renew_subs 2023-10-30T20:46:43.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.155+0000 7f532f056640 10 monclient(hunting): authenticate will time out at 2926.444601s 2023-10-30T20:46:43.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.156+0000 7f5327fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532806f0d0 0x7f532806e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:43.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.156+0000 7f532d5f3640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f532813ba30 0x7f5328141c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:38006/0 (socket says 172.21.15.69:38006) 2023-10-30T20:46:43.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.156+0000 7f532d5f3640 1 -- 172.21.15.69:0/470220230 learned_addr learned my addr 172.21.15.69:0/470220230 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-10-30T20:46:43.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.156+0000 7f532cdf2640 1 --2- 172.21.15.69:0/470220230 >> v2:172.21.15.183:3300/0 conn(0x7f532806ecf0 0x7f5328070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:43.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.156+0000 7f5327fff640 10 monclient(hunting): get_auth_request con 0x7f532806f0d0 auth_method 0 2023-10-30T20:46:43.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.156+0000 7f5327fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:43.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.156+0000 7f5327fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:43.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.156+0000 7f5327fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:43.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.156+0000 7f532cdf2640 10 monclient(hunting): get_auth_request con 0x7f532806ecf0 auth_method 0 2023-10-30T20:46:43.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.156+0000 7f532cdf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:43.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.156+0000 7f532cdf2640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:43.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.156+0000 7f532cdf2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:43.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.157+0000 7f5327fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:43.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.157+0000 7f5327fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:43.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.157+0000 7f5327fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:43.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.157+0000 7f532cdf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:43.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.157+0000 7f532cdf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:43.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.157+0000 7f532cdf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:43.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f53277fe640 1 -- 172.21.15.69:0/470220230 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532082557 0 0) 0x7f5320001490 con 0x7f532813ba30 2023-10-30T20:46:43.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f53277fe640 10 monclient(hunting): _negotiate 2023-10-30T20:46:43.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f53277fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:43.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f53277fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:43.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f53277fe640 10 monclient(hunting): my global_id is 16338 2023-10-30T20:46:43.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f53277fe640 1 -- 172.21.15.69:0/470220230 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5318003520 con 0x7f532813ba30 2023-10-30T20:46:43.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f5327fff640 10 monclient(hunting): handle_auth_done global_id 18368 payload 274 2023-10-30T20:46:43.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f5327fff640 1 -- 172.21.15.69:0/470220230 >> v2:172.21.15.183:3300/0 conn(0x7f532806ecf0 msgr2=0x7f5328070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T20:46:43.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f5327fff640 1 --2- 172.21.15.69:0/470220230 >> v2:172.21.15.183:3300/0 conn(0x7f532806ecf0 0x7f5328070860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:43.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f5327fff640 1 -- 172.21.15.69:0/470220230 >> v1:172.21.15.69:6789/0 conn(0x7f532813ba30 legacy=0x7f5328141c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:43.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f5327fff640 10 monclient: _finish_hunting 0 2023-10-30T20:46:43.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f5327fff640 1 monclient: found mon.noname-b 2023-10-30T20:46:43.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f5327fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:46:43.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f5327fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-10-30T20:46:43.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.158+0000 7f5327fff640 1 -- 172.21.15.69:0/470220230 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53280ad8f0 con 0x7f532806f0d0 2023-10-30T20:46:43.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.160+0000 7f5327fff640 1 --2- 172.21.15.69:0/470220230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532806f0d0 0x7f532806e010 secure :-1 s=READY pgs=3224 cs=0 l=1 rev1=1 crypto rx=0x7f53140097a0 tx=0x7f531402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbd9b64db53b6a6d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:43.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f53277fe640 1 -- 172.21.15.69:0/470220230 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5314031700 con 0x7f532806f0d0 2023-10-30T20:46:43.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f53277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:43.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f53277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T20:46:43.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f53277fe640 10 monclient: dump: 2023-10-30T20:46:43.173 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:43.173 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:43.174 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:43.174 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:43.174 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:43.174 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:43.174 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:43.175 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:43.175 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:43.175 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:43.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f53277fe640 10 monclient: _finish_auth 0 2023-10-30T20:46:43.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f53277fe640 10 monclient: _check_auth_tickets 2023-10-30T20:46:43.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f53277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:46:43.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f53277fe640 1 -- 172.21.15.69:0/470220230 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5314031870 con 0x7f532806f0d0 2023-10-30T20:46:43.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f53277fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:46:43.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f532f056640 5 monclient: authenticate success, global_id 18368 2023-10-30T20:46:43.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f532f056640 10 monclient: get_monmap_and_config success 2023-10-30T20:46:43.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f53277fe640 1 -- 172.21.15.69:0/470220230 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5314031c10 con 0x7f532806f0d0 2023-10-30T20:46:43.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f532f056640 4 set_mon_vals no callback set 2023-10-30T20:46:43.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f53277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:43.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f53277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:46:43.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.164+0000 7f53277fe640 10 monclient: dump: 2023-10-30T20:46:43.177 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:43.177 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:43.177 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:43.178 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:43.178 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:43.178 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:43.178 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:43.178 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:43.179 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:43.179 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:43.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.165+0000 7f532f056640 10 monclient: shutdown 2023-10-30T20:46:43.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.165+0000 7f532f056640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:46:43.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.165+0000 7f532f056640 1 -- 172.21.15.69:0/470220230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532806f0d0 msgr2=0x7f532806e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:43.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.165+0000 7f532f056640 1 --2- 172.21.15.69:0/470220230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532806f0d0 0x7f532806e010 secure :-1 s=READY pgs=3224 cs=0 l=1 rev1=1 crypto rx=0x7f53140097a0 tx=0x7f531402b7a0 comp rx=0 tx=0).stop 2023-10-30T20:46:43.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.165+0000 7f532f056640 1 -- 172.21.15.69:0/470220230 shutdown_connections 2023-10-30T20:46:43.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.165+0000 7f532f056640 1 --2- 172.21.15.69:0/470220230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532806f0d0 0x7f532806e010 unknown :-1 s=CLOSED pgs=3224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:43.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.165+0000 7f532f056640 1 --2- 172.21.15.69:0/470220230 >> v2:172.21.15.183:3300/0 conn(0x7f532806ecf0 0x7f5328070860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:43.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.165+0000 7f532f056640 1 -- 172.21.15.69:0/470220230 >> 172.21.15.69:0/470220230 conn(0x7f532806a7a0 msgr2=0x7f53280b35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:46:43.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.165+0000 7f532f056640 1 -- 172.21.15.69:0/470220230 shutdown_connections 2023-10-30T20:46:43.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.165+0000 7f532f056640 1 -- 172.21.15.69:0/470220230 wait complete. 2023-10-30T20:46:43.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.165+0000 7f532f056640 10 monclient: build_initial_monmap 2023-10-30T20:46:43.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.165+0000 7f532f056640 10 monclient: monmap: 2023-10-30T20:46:43.181 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-10-30T20:46:43.181 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T20:46:43.181 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-10-30T20:46:43.182 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-10-30T20:46:43.182 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-10-30T20:46:43.182 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-10-30T20:46:43.182 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-10-30T20:46:43.182 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.noname-b 2023-10-30T20:46:43.182 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-10-30T20:46:43.183 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:43.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 1 librados: starting msgr at 2023-10-30T20:46:43.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 1 librados: starting objecter 2023-10-30T20:46:43.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 1 Processor -- start 2023-10-30T20:46:43.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 1 -- start start 2023-10-30T20:46:43.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 1 librados: setting wanted keys 2023-10-30T20:46:43.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 1 librados: calling monclient init 2023-10-30T20:46:43.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 10 monclient: init 2023-10-30T20:46:43.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 10 monclient: _reopen_session rank -1 2023-10-30T20:46:43.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T20:46:43.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f532806ecf0 0x7f532813d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:43.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 10 monclient(hunting): picked mon.noname-b con 0x7f532806ecf0 addr [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-10-30T20:46:43.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532806f0d0 0x7f532813f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:43.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 10 monclient(hunting): picked mon.noname-a con 0x7f532806f0d0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-10-30T20:46:43.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532813ba30 0x7f53281c8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:43.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 10 monclient(hunting): picked mon.noname-c con 0x7f532813ba30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-10-30T20:46:43.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:43.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5328070b50 con 0x7f532806f0d0 2023-10-30T20:46:43.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:43.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532813e960 con 0x7f532806ecf0 2023-10-30T20:46:43.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 10 monclient(hunting): start opening mon connection 2023-10-30T20:46:43.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532813cfe0 con 0x7f532813ba30 2023-10-30T20:46:43.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 10 monclient(hunting): _renew_subs 2023-10-30T20:46:43.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.166+0000 7f532f056640 10 monclient(hunting): authenticate will time out at 2926.455703s 2023-10-30T20:46:43.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f5327fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532806f0d0 0x7f532813f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:43.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f532d5f3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532813ba30 0x7f53281c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:43.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f532cdf2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f532806ecf0 0x7f532813d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:43.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f5327fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532806f0d0 0x7f532813f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34282/0 (socket says 172.21.15.69:34282) 2023-10-30T20:46:43.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f532d5f3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532813ba30 0x7f53281c8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:51634/0 (socket says 172.21.15.69:51634) 2023-10-30T20:46:43.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f5327fff640 1 -- 172.21.15.69:0/1746544416 learned_addr learned my addr 172.21.15.69:0/1746544416 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-10-30T20:46:43.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f5327fff640 10 monclient(hunting): get_auth_request con 0x7f532806f0d0 auth_method 0 2023-10-30T20:46:43.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f5327fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:43.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f5327fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:43.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f5327fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:43.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f532d5f3640 10 monclient(hunting): get_auth_request con 0x7f532813ba30 auth_method 0 2023-10-30T20:46:43.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f532d5f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:43.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f532d5f3640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:43.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f532d5f3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:43.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f532cdf2640 10 monclient(hunting): get_auth_request con 0x7f532806ecf0 auth_method 0 2023-10-30T20:46:43.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f532cdf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T20:46:43.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f532cdf2640 10 monclient(hunting): _init_auth method 2 2023-10-30T20:46:43.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.167+0000 7f532cdf2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T20:46:43.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.168+0000 7f5327fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:43.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.168+0000 7f5327fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:43.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.168+0000 7f5327fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:43.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.168+0000 7f532d5f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:43.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.168+0000 7f532d5f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:43.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.168+0000 7f532d5f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:43.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.168+0000 7f532cdf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T20:46:43.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.168+0000 7f532cdf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T20:46:43.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.168+0000 7f532cdf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T20:46:43.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.171+0000 7f532d5f3640 10 monclient(hunting): handle_auth_done global_id 18371 payload 824 2023-10-30T20:46:43.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.171+0000 7f532d5f3640 1 -- 172.21.15.69:0/1746544416 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f532806ecf0 msgr2=0x7f532813d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:43.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.171+0000 7f532d5f3640 1 --2- 172.21.15.69:0/1746544416 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f532806ecf0 0x7f532813d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:43.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.171+0000 7f532d5f3640 1 -- 172.21.15.69:0/1746544416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532806f0d0 msgr2=0x7f532813f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:43.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.171+0000 7f532d5f3640 1 --2- 172.21.15.69:0/1746544416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532806f0d0 0x7f532813f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:43.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.171+0000 7f532d5f3640 10 monclient: _finish_hunting 0 2023-10-30T20:46:43.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.171+0000 7f532d5f3640 1 monclient: found mon.noname-c 2023-10-30T20:46:43.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.171+0000 7f532d5f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T20:46:43.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.171+0000 7f532d5f3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-10-30T20:46:43.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.171+0000 7f532d5f3640 1 -- 172.21.15.69:0/1746544416 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532813d240 con 0x7f532813ba30 2023-10-30T20:46:43.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.171+0000 7f5327fff640 1 --2- 172.21.15.69:0/1746544416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532806f0d0 0x7f532813f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T20:46:43.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.173+0000 7f532d5f3640 1 --2- 172.21.15.69:0/1746544416 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532813ba30 0x7f53281c8500 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 crypto rx=0x7f532000caa0 tx=0x7f532000cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:43.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.173+0000 7f53257fa640 10 client.?.objecter ms_handle_connect 0x7f532813ba30 2023-10-30T20:46:43.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.173+0000 7f53257fa640 10 client.?.objecter resend_mon_ops 2023-10-30T20:46:43.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f53257fa640 1 -- 172.21.15.69:0/1746544416 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532000aba0 con 0x7f532813ba30 2023-10-30T20:46:43.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f53257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:43.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f53257fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T20:46:43.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f53257fa640 10 monclient: dump: 2023-10-30T20:46:43.197 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:43.197 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:43.197 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:43.197 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:43.197 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:43.198 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:43.198 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:43.198 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:43.198 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:43.198 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:43.198 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f53257fa640 10 monclient: _finish_auth 0 2023-10-30T20:46:43.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f53257fa640 10 monclient: _check_auth_tickets 2023-10-30T20:46:43.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f53257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T20:46:43.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f53257fa640 1 -- 172.21.15.69:0/1746544416 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532000e040 con 0x7f532813ba30 2023-10-30T20:46:43.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f532f056640 5 monclient: authenticate success, global_id 18371 2023-10-30T20:46:43.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f532f056640 10 monclient: _renew_subs 2023-10-30T20:46:43.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f532f056640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:46:43.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f532f056640 1 -- 172.21.15.69:0/1746544416 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532813f830 con 0x7f532813ba30 2023-10-30T20:46:43.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f53257fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T20:46:43.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f53257fa640 1 -- 172.21.15.69:0/1746544416 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5320012400 con 0x7f532813ba30 2023-10-30T20:46:43.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f53257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T20:46:43.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f53257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T20:46:43.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f53257fa640 10 monclient: dump: 2023-10-30T20:46:43.201 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-10-30T20:46:43.201 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 32c1a723-7694-47d8-8b37-b1257dfe1796 2023-10-30T20:46:43.201 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:43.201 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-10-30T20:22:05.015181+0000 2023-10-30T20:46:43.201 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 18 (reef) 2023-10-30T20:46:43.202 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2023-10-30T20:46:43.202 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-10-30T20:46:43.202 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2023-10-30T20:46:43.202 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-10-30T20:46:43.202 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:46:43.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f5326ffd640 4 set_mon_vals no callback set 2023-10-30T20:46:43.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f532f056640 10 client.18371.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T20:46:43.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f532f056640 10 monclient: _renew_subs 2023-10-30T20:46:43.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f532f056640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:46:43.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f532f056640 1 -- 172.21.15.69:0/1746544416 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532813c560 con 0x7f532813ba30 2023-10-30T20:46:43.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.177+0000 7f532f056640 1 librados: init done 2023-10-30T20:46:43.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.178+0000 7f532f056640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:43.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.178+0000 7f532f056640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:43.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.178+0000 7f532f056640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:46:43.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.178+0000 7f532f056640 1 -- 172.21.15.69:0/1746544416 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f532813d440 con 0x7f532813ba30 2023-10-30T20:46:43.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.196+0000 7f53257fa640 1 -- 172.21.15.69:0/1746544416 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f5320007070 con 0x7f532813ba30 2023-10-30T20:46:43.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.197+0000 7f53257fa640 1 --2- 172.21.15.69:0/1746544416 >> v2:172.21.15.69:6800/1876307353 conn(0x7f530c06c7c0 0x7f530c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T20:46:43.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.197+0000 7f532cdf2640 1 --2- 172.21.15.69:0/1746544416 >> v2:172.21.15.69:6800/1876307353 conn(0x7f530c06c7c0 0x7f530c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T20:46:43.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.197+0000 7f532cdf2640 10 monclient: get_auth_request con 0x7f530c06c7c0 auth_method 0 2023-10-30T20:46:43.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.197+0000 7f532cdf2640 1 --2- 172.21.15.69:0/1746544416 >> v2:172.21.15.69:6800/1876307353 conn(0x7f530c06c7c0 0x7f530c06ec80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f531c009b40 tx=0x7f531c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T20:46:43.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.197+0000 7f53257fa640 10 client.18371.objecter ms_handle_connect 0x7f530c06c7c0 2023-10-30T20:46:43.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.200+0000 7f53257fa640 1 -- 172.21.15.69:0/1746544416 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(904..904 src has 451..904) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f53200580a0 con 0x7f532813ba30 2023-10-30T20:46:43.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.200+0000 7f53257fa640 10 client.18371.objecter ms_dispatch 0x7f5328000b90 osd_map(904..904 src has 451..904) v4 2023-10-30T20:46:43.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.200+0000 7f53257fa640 3 client.18371.objecter handle_osd_map got epochs [904,904] > 0 2023-10-30T20:46:43.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.200+0000 7f53257fa640 3 client.18371.objecter handle_osd_map decoding full epoch 904 2023-10-30T20:46:43.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.200+0000 7f53257fa640 20 client.18371.objecter dump_active .. 0 homeless 2023-10-30T20:46:43.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.431+0000 7f53257fa640 1 -- 172.21.15.69:0/1746544416 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f532005c140 con 0x7f532813ba30 2023-10-30T20:46:43.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.431+0000 7f53257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T20:46:43.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.431+0000 7f53257fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T20:46:43.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.604+0000 7f532f056640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T20:46:43.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.604+0000 7f532f056640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T20:46:43.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.604+0000 7f532f056640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-10-30T20:46:43.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.604+0000 7f532f056640 1 -- 172.21.15.69:0/1746544416 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f532813f830 con 0x7f532813ba30 2023-10-30T20:46:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.808+0000 7f53257fa640 1 -- 172.21.15.69:0/1746544416 <== mon.2 v2:172.21.15.69:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f532005ce80 con 0x7f532813ba30 2023-10-30T20:46:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.808+0000 7f53257fa640 10 monclient: handle_config config(1 keys) v1 2023-10-30T20:46:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.808+0000 7f5326ffd640 4 set_mon_vals no callback set 2023-10-30T20:46:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.808+0000 7f5326ffd640 10 set_mon_vals mon_health_to_clog = false 2023-10-30T20:46:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.808+0000 7f53257fa640 1 -- 172.21.15.69:0/1746544416 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f532005e160 con 0x7f532813ba30 2023-10-30T20:46:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.808+0000 7f53257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T20:46:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.808+0000 7f53257fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T20:46:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.811+0000 7f532f056640 10 librados: watch_flush enter 2023-10-30T20:46:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.811+0000 7f532f056640 10 librados: watch_flush exit 2023-10-30T20:46:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.811+0000 7f532f056640 20 client.18371.objecter shutdown clearing up homeless session... 2023-10-30T20:46:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.811+0000 7f532f056640 10 client.18371.objecter successfully canceled tick 2023-10-30T20:46:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.811+0000 7f532f056640 1 -- 172.21.15.69:0/1746544416 >> v2:172.21.15.69:6800/1876307353 conn(0x7f530c06c7c0 msgr2=0x7f530c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.811+0000 7f532f056640 1 --2- 172.21.15.69:0/1746544416 >> v2:172.21.15.69:6800/1876307353 conn(0x7f530c06c7c0 0x7f530c06ec80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f531c009b40 tx=0x7f531c009190 comp rx=0 tx=0).stop 2023-10-30T20:46:43.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.811+0000 7f532f056640 10 monclient: shutdown 2023-10-30T20:46:43.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.811+0000 7f532f056640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T20:46:43.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.811+0000 7f532f056640 1 -- 172.21.15.69:0/1746544416 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532813ba30 msgr2=0x7f53281c8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T20:46:43.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.811+0000 7f532f056640 1 --2- 172.21.15.69:0/1746544416 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532813ba30 0x7f53281c8500 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 crypto rx=0x7f532000caa0 tx=0x7f532000cf60 comp rx=0 tx=0).stop 2023-10-30T20:46:43.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.811+0000 7f532f056640 1 -- 172.21.15.69:0/1746544416 shutdown_connections 2023-10-30T20:46:43.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.811+0000 7f532f056640 1 --2- 172.21.15.69:0/1746544416 >> v2:172.21.15.69:6800/1876307353 conn(0x7f530c06c7c0 0x7f530c06ec80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.811+0000 7f532f056640 1 --2- 172.21.15.69:0/1746544416 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f532813ba30 0x7f53281c8500 unknown :-1 s=CLOSED pgs=3225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.812+0000 7f532f056640 1 --2- 172.21.15.69:0/1746544416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f532806f0d0 0x7f532813f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.812+0000 7f532f056640 1 --2- 172.21.15.69:0/1746544416 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f532806ecf0 0x7f532813d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T20:46:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.812+0000 7f532f056640 1 -- 172.21.15.69:0/1746544416 >> 172.21.15.69:0/1746544416 conn(0x7f532806a7a0 msgr2=0x7f532813a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T20:46:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.812+0000 7f532f056640 1 -- 172.21.15.69:0/1746544416 shutdown_connections 2023-10-30T20:46:43.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.812+0000 7f532f056640 1 -- 172.21.15.69:0/1746544416 wait complete. 2023-10-30T20:46:43.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-10-30T20:46:43.812+0000 7f532f056640 1 librados: shutdown 2023-10-30T20:46:43.828 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-30T20:46:43.828 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-30T20:46:43.829 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-30T20:46:43.829 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T20:46:43.847 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-10-30T20:46:43.832+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86370) UID: 0 2023-10-30T20:46:43.847 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-10-30T20:46:43.834+0000 ac7d640 -1 osd.0 904 *** Got signal Terminated *** 2023-10-30T20:46:48.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-30T20:46:49.931 INFO:tasks.ceph.osd.0:Stopped 2023-10-30T20:46:49.931 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-30T20:46:49.932 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T20:46:49.946 INFO:tasks.ceph.osd.1.smithi069.stderr:2023-10-30T20:46:49.936+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86374) UID: 0 2023-10-30T20:46:49.946 INFO:tasks.ceph.osd.1.smithi069.stderr:2023-10-30T20:46:49.938+0000 ac7d640 -1 osd.1 907 *** Got signal Terminated *** 2023-10-30T20:46:50.569 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:12.628 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:50.619 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:12.646 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:50.620 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:12.647 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:50.625 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:12.684 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:50.637 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:12.664 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:50.649 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:12.708 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:50.672 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:12.731 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:50.680 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:12.739 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:50.705 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:12.764 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:46:54.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-30T20:47:10.569 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:32.628 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:10.571 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:32.629 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:10.597 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:32.656 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:10.617 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:32.644 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:10.624 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:32.651 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:10.639 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:32.698 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:10.640 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:32.700 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:10.646 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:32.673 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:10.660 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:32.719 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:11.262 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:33.289 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:11.263 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:33.290 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:11.270 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:33.297 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:11.306 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:33.365 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:11.307 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:33.366 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:11.314 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:33.373 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:11.328 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:33.387 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:11.329 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:33.388 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:11.337 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:33.396 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:16.954 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:39.013 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:16.958 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:39.017 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:16.973 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:39.031 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:17.000 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:39.028 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:17.005 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:39.032 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:17.008 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:39.067 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:17.009 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:39.068 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:17.018 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:39.046 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:17.023 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:39.082 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:30.646 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:52.704 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:30.653 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:52.711 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:30.674 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:52.733 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:30.699 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:52.726 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:30.700 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:52.759 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:30.706 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:52.765 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:30.718 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:52.745 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:30.724 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:52.783 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:30.735 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:52.762 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:31.347 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:53.374 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:31.348 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:53.375 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:31.357 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:24:53.384 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:31.386 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:53.443 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:31.386 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:53.445 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:31.387 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:53.446 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:31.388 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:53.447 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:31.397 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:53.456 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:31.397 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:53.456 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:50.821 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:12.880 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:50.825 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:12.884 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:50.839 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:12.897 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:50.877 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:12.904 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:50.879 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:12.906 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:50.892 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:12.920 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:50.895 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:12.954 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:50.897 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:12.956 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:50.914 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:12.973 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:55.663 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:17.722 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:55.711 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:17.770 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:55.724 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:17.783 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:55.757 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:17.784 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:55.761 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:17.820 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:55.762 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:17.789 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:55.764 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:17.823 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:55.773 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:17.801 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:47:55.776 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:17.835 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:09.248 INFO:tasks.ceph.osd.1:Stopped 2023-10-30T20:48:09.249 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-30T20:48:09.249 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T20:48:09.261 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-10-30T20:48:09.252+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86386) UID: 0 2023-10-30T20:48:09.264 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-10-30T20:48:09.253+0000 ac7d640 -1 osd.2 910 *** Got signal Terminated *** 2023-10-30T20:48:10.867 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:32.925 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:10.868 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:32.927 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:10.895 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:32.953 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:10.918 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:32.976 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:10.920 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:32.979 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:10.926 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:32.954 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:10.931 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:32.959 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:10.938 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:32.997 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:10.948 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:32.975 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:14.824 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-30T20:48:15.352 INFO:tasks.ceph.osd.2:Stopped 2023-10-30T20:48:15.352 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-30T20:48:15.352 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T20:48:15.368 INFO:tasks.ceph.osd.3.smithi069.stderr:2023-10-30T20:48:15.355+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86394) UID: 0 2023-10-30T20:48:15.368 INFO:tasks.ceph.osd.3.smithi069.stderr:2023-10-30T20:48:15.356+0000 ac7d640 -1 osd.3 912 *** Got signal Terminated *** 2023-10-30T20:48:15.813 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:37.872 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:15.881 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:37.939 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:15.908 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:37.966 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:15.923 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:37.950 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:15.928 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:37.986 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:15.946 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:37.973 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:15.957 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:38.017 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:15.958 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:37.985 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:15.980 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:38.039 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:20.832 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-30T20:48:25.798 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:47.856 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:25.803 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:47.862 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:25.823 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:47.881 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:25.849 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:47.908 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:25.852 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:47.879 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:25.853 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:47.911 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:25.856 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:47.883 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:25.873 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:47.900 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:25.883 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:47.942 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:35.827 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:57.885 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:35.832 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:57.890 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:35.847 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:57.905 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:35.880 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:57.939 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:35.881 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:57.908 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:35.884 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:57.911 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:35.885 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:57.943 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:35.897 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:57.956 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:35.902 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:25:57.929 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:45.856 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:07.914 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:45.858 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:07.917 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:45.872 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:07.930 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:45.906 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:07.965 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:45.908 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:07.935 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:45.910 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:07.937 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:45.910 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:07.969 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:45.921 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:07.949 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:45.927 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:07.986 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:51.960 INFO:tasks.ceph.osd.3:Stopped 2023-10-30T20:48:51.961 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-30T20:48:51.961 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T20:48:51.974 INFO:tasks.ceph.osd.4.smithi183.stderr:2023-10-30T20:48:51.964+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86328) UID: 0 2023-10-30T20:48:51.974 INFO:tasks.ceph.osd.4.smithi183.stderr:2023-10-30T20:48:51.965+0000 ac7d640 -1 osd.4 916 *** Got signal Terminated *** 2023-10-30T20:48:55.886 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:17.945 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:55.888 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:17.946 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:55.901 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:17.960 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:55.934 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:17.993 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:55.937 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:17.996 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:55.940 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:17.967 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:55.942 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:17.969 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:55.950 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:18.009 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:55.955 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:17.983 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:48:56.679 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-10-30T20:48:58.062 INFO:tasks.ceph.osd.4:Stopped 2023-10-30T20:48:58.063 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-30T20:48:58.064 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T20:48:58.076 INFO:tasks.ceph.osd.5.smithi183.stderr:2023-10-30T20:48:58.066+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86331) UID: 0 2023-10-30T20:48:58.077 INFO:tasks.ceph.osd.5.smithi183.stderr:2023-10-30T20:48:58.068+0000 ac7d640 -1 osd.5 918 *** Got signal Terminated *** 2023-10-30T20:49:00.897 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:22.956 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:00.902 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:22.960 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:00.915 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:22.973 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:00.948 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:22.975 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:00.951 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:22.978 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:00.951 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:23.010 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:00.954 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:23.013 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:00.962 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:22.989 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:00.966 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:23.025 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:02.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-10-30T20:49:15.938 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:37.997 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:15.942 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:38.001 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:15.964 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:38.023 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:15.992 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:38.019 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:15.996 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:38.023 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:16.001 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:38.060 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:16.003 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:38.061 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:16.010 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:38.037 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:16.019 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:38.077 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:20.954 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:43.013 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:20.959 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:43.017 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:20.975 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:43.033 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:21.003 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:43.030 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:21.006 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:43.034 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:21.017 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:43.044 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:21.023 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:43.082 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:21.031 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:43.090 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:21.042 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:43.101 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:35.995 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:58.053 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:35.997 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:58.056 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:36.014 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:58.073 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:36.041 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:58.068 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:36.049 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:58.076 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:36.051 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:58.110 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:36.053 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:58.112 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:36.063 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:26:58.090 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:36.067 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:58.126 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:41.014 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:03.072 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:41.015 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:03.074 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:41.025 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:03.084 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:41.060 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:03.087 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:41.062 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:03.090 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:41.073 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:03.100 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:41.076 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:03.135 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:41.080 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:03.139 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:41.094 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:03.152 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:56.052 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:18.110 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:56.059 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:18.117 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:56.075 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:18.134 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:56.101 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:18.128 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:56.108 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:18.135 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:56.109 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:18.168 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:56.115 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:18.173 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:56.122 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:18.149 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:56.129 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:18.188 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:49:59.078 INFO:tasks.ceph.osd.5:Stopped 2023-10-30T20:49:59.079 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-30T20:49:59.079 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T20:49:59.096 INFO:tasks.ceph.osd.6.smithi183.stderr:2023-10-30T20:49:59.082+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86330) UID: 0 2023-10-30T20:49:59.096 INFO:tasks.ceph.osd.6.smithi183.stderr:2023-10-30T20:49:59.083+0000 ac7d640 -1 osd.6 920 *** Got signal Terminated *** 2023-10-30T20:50:01.071 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:23.129 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:01.074 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:23.132 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:01.087 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:23.145 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:01.118 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:23.145 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:01.118 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:23.177 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:01.123 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:23.150 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:01.125 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:23.184 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:01.134 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:23.161 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:01.141 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:23.200 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:16.117 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:38.175 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:16.120 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:38.179 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:16.135 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:38.194 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:16.166 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:38.193 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:16.169 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:38.228 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:16.169 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:38.197 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:16.173 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:38.232 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:16.185 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:38.212 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:16.190 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:38.249 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.125 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:43.184 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.131 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:43.190 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.145 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:43.204 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.178 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:43.205 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.180 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:43.239 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.181 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:43.208 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.184 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:43.243 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.191 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:43.218 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.195 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:43.254 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.926 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:43.953 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.934 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:43.993 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.955 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:44.014 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.956 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:44.015 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.967 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:44.026 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.973 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:44.000 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.981 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:44.040 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.989 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:44.016 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:21.994 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:44.053 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:36.172 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:58.230 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:36.175 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:58.233 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:36.188 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:58.247 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:36.223 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:58.251 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:36.225 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:58.252 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:36.239 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:27:58.266 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:36.243 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:58.303 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:36.245 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:58.304 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:36.259 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:58.318 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:41.180 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:03.238 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:41.186 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:03.245 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:41.201 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:03.260 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:41.231 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:03.258 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:41.235 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:03.262 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:41.246 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:03.273 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:41.261 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:03.319 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:41.263 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:03.322 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:41.278 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:03.337 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:56.225 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:18.283 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:56.229 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:18.288 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:56.246 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:18.304 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:56.272 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:18.299 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:56.276 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:18.303 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:56.279 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:18.338 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:56.285 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:18.344 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:56.292 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:18.319 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:50:56.302 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:18.361 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:01.241 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:23.268 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:01.242 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:23.269 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:01.243 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:23.302 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:01.245 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:23.304 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:01.247 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:23.306 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:01.248 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:23.307 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:01.254 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:23.282 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:01.262 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:23.321 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:01.266 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:23.325 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:16.404 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:38.461 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:16.406 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:38.464 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:16.419 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:38.478 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:16.452 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:38.479 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:16.452 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:38.511 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:16.453 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:38.480 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:16.454 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:38.512 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:16.468 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:38.495 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:16.474 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:38.533 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:20.997 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:20.991+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:21.314 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:43.372 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:21.315 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:43.374 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:21.326 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:43.384 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:21.356 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:43.383 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:21.362 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:43.389 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:21.374 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:43.401 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:21.379 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:43.439 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:21.381 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:43.440 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:21.391 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:43.450 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:22.031 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:22.028+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:23.004 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:23.002+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:23.998 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:23.996+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:25.047 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:25.042+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:26.020 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:26.018+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:27.002 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:27.000+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:28.045 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:28.043+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:29.019 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:29.018+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:30.046 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:30.044+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:31.052 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:31.051+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:32.049 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:32.048+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:33.057 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:33.056+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:34.013 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:34.011+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:34.979 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:34.977+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:35.991 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:35.989+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:36.354 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:58.413 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:36.363 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:58.422 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:36.381 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:58.440 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:36.408 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:58.435 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:36.415 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:58.474 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:36.416 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:58.442 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:36.419 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:58.478 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:36.433 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:28:58.460 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:36.436 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:58.495 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:37.032 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:37.030+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:38.039 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:38.037+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:39.089 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:39.087+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:40.057 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:40.055+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:41.065 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:41.064+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:41.366 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:03.424 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:41.373 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:03.432 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:41.385 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:03.444 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:41.418 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:03.445 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:41.419 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:03.478 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:41.421 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:03.449 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:41.423 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:03.482 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:41.433 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:03.460 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:41.434 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:03.494 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:42.103 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:42.101+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:43.105 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:43.103+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:44.065 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:44.063+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:45.031 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:45.030+0000 fa59640 -1 osd.7 920 heartbeat_check: no reply from 172.21.15.183:6806 osd.6 since back 2023-10-30T20:49:56.461911+0000 front 2023-10-30T20:49:56.463316+0000 (oldest deadline 2023-10-30T20:51:20.560952+0000) 2023-10-30T20:51:48.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-10-30T20:51:48.903 INFO:tasks.ceph.osd.6:Stopped 2023-10-30T20:51:48.904 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-30T20:51:48.904 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T20:51:48.916 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:48.907+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86335) UID: 0 2023-10-30T20:51:48.916 INFO:tasks.ceph.osd.7.smithi183.stderr:2023-10-30T20:51:48.908+0000 ac7d640 -1 osd.7 920 *** Got signal Terminated *** 2023-10-30T20:51:54.361 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-10-30T20:51:56.418 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:18.477 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:56.422 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:18.480 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:56.440 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:18.499 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:56.468 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:18.495 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:56.472 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:18.499 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:56.485 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:18.545 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:56.493 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:18.552 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:56.495 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:18.523 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:51:56.509 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:18.568 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:01.432 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:23.490 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:01.433 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:23.492 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:01.443 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:23.501 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:01.476 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:23.534 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:01.477 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:23.536 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:01.480 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:23.508 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:01.482 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:23.510 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:01.489 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:23.548 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:01.497 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:23.524 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:16.469 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:38.528 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:16.474 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:38.533 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:16.491 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:38.549 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:16.521 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:38.581 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:16.523 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:38.551 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:16.525 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:38.584 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:16.526 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:38.553 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:16.541 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:38.569 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:16.542 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:38.601 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:21.486 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:43.544 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:21.493 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:43.551 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:21.506 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:43.564 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:21.539 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:43.567 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:21.544 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:43.571 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:21.554 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:43.581 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:21.555 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:43.614 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:21.558 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:43.617 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:21.569 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:43.628 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:36.538 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:58.596 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:36.542 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:58.601 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:36.558 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:58.616 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:36.586 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:58.613 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:36.590 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:58.617 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:36.593 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:58.652 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:36.595 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:58.654 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:36.606 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:29:58.633 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:36.611 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:58.670 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:41.550 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:03.605 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:41.554 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:03.612 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:41.569 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:03.628 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:41.602 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:03.629 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:41.604 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:03.631 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:41.607 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:03.666 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:41.610 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:03.669 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:41.615 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:03.643 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:41.621 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:03.681 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:56.594 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:18.652 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:56.597 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:18.656 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:56.622 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:18.681 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:56.645 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:18.672 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:56.656 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:18.715 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:56.659 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:18.718 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:56.665 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:18.692 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:56.676 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:18.735 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:52:56.683 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:18.710 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:01.608 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:23.666 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:01.615 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:23.674 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:01.630 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:23.689 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:01.660 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:23.719 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:01.663 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:23.690 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:01.667 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:23.695 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:01.668 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:23.727 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:01.680 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:23.707 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:01.682 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:23.741 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:16.673 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:38.732 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:16.679 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:38.737 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:16.703 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:38.761 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:16.726 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:38.753 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:16.731 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:38.790 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:16.735 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:38.794 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:16.744 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:38.771 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:16.753 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:38.812 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:16.760 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:38.787 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:21.701 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:43.759 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:21.731 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:43.790 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:21.743 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:43.802 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:21.747 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:43.807 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:21.749 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:43.776 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:21.751 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:43.810 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:21.753 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:43.781 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:21.791 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:43.818 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:21.792 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:43.851 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:33.007 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:55.065 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:33.012 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:55.070 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:33.032 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:55.091 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:33.060 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:55.087 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:33.062 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:55.089 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:33.066 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:55.125 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:33.068 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:55.127 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:33.081 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:55.108 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:33.087 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:55.146 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:36.736 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:58.795 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:36.739 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:58.798 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:36.748 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:58.807 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:36.789 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:58.848 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:36.791 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:58.850 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:36.795 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:58.822 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:36.797 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:58.824 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:36.802 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:58.861 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:36.805 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:30:58.833 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:41.705 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:03.764 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:41.752 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:03.779 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:41.753 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:03.780 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:41.757 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:03.816 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:41.765 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:03.792 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:41.772 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:03.831 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:41.803 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:03.862 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:41.808 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:03.867 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:53:41.823 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:03.882 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:01.879 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:23.938 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:01.883 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:23.942 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:01.900 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:23.959 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:01.930 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:23.957 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:01.932 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:23.960 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:01.933 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:23.992 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:01.934 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:23.993 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:01.949 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:23.977 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:01.951 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:24.010 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:06.898 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:28.957 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:06.901 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:28.960 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:06.915 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:28.973 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:06.947 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:28.974 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:06.954 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:29.013 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:06.955 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:28.983 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:06.957 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:29.016 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:06.966 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:28.994 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:06.968 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:29.027 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:21.906 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:43.965 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:21.908 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:43.967 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:21.925 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:43.983 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:21.955 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:43.982 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:21.959 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:44.018 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:21.960 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:43.986 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:21.960 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:44.019 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:21.976 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:44.003 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:21.978 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:44.037 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:26.923 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:48.982 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:26.927 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:48.986 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:26.939 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:48.998 86007== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:26.971 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:49.030 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:26.974 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:49.001 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:26.976 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:49.003 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:26.976 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:49.035 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:26.988 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:31:49.015 85794== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:26.988 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:49.047 86011== Warning: unimplemented fcntl command: 1036 2023-10-30T20:54:27.541 INFO:tasks.ceph.osd.7:Stopped 2023-10-30T20:54:27.542 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-30T20:54:27.542 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-30T20:54:27.543 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T20:54:27.543 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-10-30T20:54:27.542+0000 7fea6cb52640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 86019) UID: 0 2023-10-30T20:54:27.569 INFO:tasks.ceph.mgr.y:Stopped 2023-10-30T20:54:27.570 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-30T20:54:27.570 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T20:54:27.593 INFO:tasks.ceph.mgr.x:Stopped 2023-10-30T20:54:27.593 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-30T20:54:27.594 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T20:54:27.594 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T20:54:27.624 INFO:tasks.ceph.mon.a.smithi069.stderr:2023-10-30T20:54:27.613+0000 9f5f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 85979) UID: 0 2023-10-30T20:54:27.624 INFO:tasks.ceph.mon.a.smithi069.stderr:2023-10-30T20:54:27.614+0000 9f5f640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-10-30T20:54:28.209 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:50.268 86007== 2023-10-30T20:54:28.209 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:50.268 86007== Exit program on first error (--exit-on-first-error=yes) 2023-10-30T20:54:28.226 INFO:tasks.ceph.mon.a.smithi069.stderr:daemon-helper: command failed with exit status 42 2023-10-30T20:54:30.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-10-30T20:54:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-10-30T20:54:33.697 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-10-30T20:54:33.698 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T20:54:33.698 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T20:54:33.699 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T20:54:33.714 INFO:tasks.ceph.mon.c.smithi069.stderr:2023-10-30T20:54:33.702+0000 9f5f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 85981) UID: 0 2023-10-30T20:54:33.715 INFO:tasks.ceph.mon.c.smithi069.stderr:2023-10-30T20:54:33.703+0000 9f5f640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-10-30T20:54:34.389 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:56.448 86011== 2023-10-30T20:54:34.390 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:56.448 86011== Exit program on first error (--exit-on-first-error=yes) 2023-10-30T20:54:34.406 INFO:tasks.ceph.mon.c.smithi069.stderr:daemon-helper: command failed with exit status 42 2023-10-30T20:54:36.005 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-10-30T20:54:36.006 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-10-30T20:54:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-10-30T20:54:39.801 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-10-30T20:54:39.801 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T20:54:39.801 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T20:54:39.802 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T20:54:39.816 INFO:tasks.ceph.mon.b.smithi183.stderr:2023-10-30T20:54:39.805+0000 9f5f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 85790) UID: 0 2023-10-30T20:54:39.817 INFO:tasks.ceph.mon.b.smithi183.stderr:2023-10-30T20:54:39.806+0000 9f5f640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2023-10-30T20:54:40.567 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:32:02.594 85794== 2023-10-30T20:54:40.568 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:32:02.594 85794== Exit program on first error (--exit-on-first-error=yes) 2023-10-30T20:54:40.583 INFO:tasks.ceph.mon.b.smithi183.stderr:daemon-helper: command failed with exit status 42 2023-10-30T20:54:41.007 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-10-30T20:54:41.007 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-10-30T20:54:45.903 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-10-30T20:54:45.904 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi183 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-10-30T20:54:45.904 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T20:54:45.905 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-30T20:54:45.905 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-10-30T20:54:45.944 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi069.front.sepia.ceph.com 2023-10-30T20:54:45.944 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-30T20:54:46.007 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-10-30T20:54:46.122 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi069.front.sepia.ceph.com 2023-10-30T20:54:46.122 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-30T20:54:46.196 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi069.front.sepia.ceph.com 2023-10-30T20:54:46.196 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-30T20:54:46.264 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi069.front.sepia.ceph.com 2023-10-30T20:54:46.264 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-30T20:54:46.318 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi183.front.sepia.ceph.com 2023-10-30T20:54:46.319 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-30T20:54:46.440 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi183.front.sepia.ceph.com 2023-10-30T20:54:46.441 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-30T20:54:46.503 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi183.front.sepia.ceph.com 2023-10-30T20:54:46.503 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-30T20:54:46.550 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi183.front.sepia.ceph.com 2023-10-30T20:54:46.550 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-30T20:54:46.601 INFO:tasks.ceph:Archiving mon data... 2023-10-30T20:54:46.602 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403/data/mon.a.tgz 2023-10-30T20:54:46.602 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2023-10-30T20:54:46.619 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.FsKX9bTz7G 2023-10-30T20:54:46.619 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.FsKX9bTz7G 2023-10-30T20:54:46.788 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.FsKX9bTz7G 2023-10-30T20:54:46.872 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.FsKX9bTz7G is 454KB 2023-10-30T20:54:46.969 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.FsKX9bTz7G 2023-10-30T20:54:46.983 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403/data/mon.c.tgz 2023-10-30T20:54:46.983 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2023-10-30T20:54:47.037 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.g9Rt0en2dz 2023-10-30T20:54:47.037 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.g9Rt0en2dz 2023-10-30T20:54:47.183 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.g9Rt0en2dz 2023-10-30T20:54:47.266 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.g9Rt0en2dz is 454KB 2023-10-30T20:54:47.334 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.g9Rt0en2dz 2023-10-30T20:54:47.348 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403/data/mon.b.tgz 2023-10-30T20:54:47.348 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2023-10-30T20:54:47.366 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.vbJpVZkLmx 2023-10-30T20:54:47.366 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.vbJpVZkLmx 2023-10-30T20:54:47.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.vbJpVZkLmx 2023-10-30T20:54:47.621 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.vbJpVZkLmx is 454KB 2023-10-30T20:54:47.689 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.vbJpVZkLmx 2023-10-30T20:54:47.703 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-30T20:54:47.704 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T20:54:47.706 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T20:54:47.770 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-30T20:54:47.770 DEBUG:teuthology.orchestra.run.smithi069:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T20:54:47.773 DEBUG:teuthology.orchestra.run.smithi183:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T20:54:47.813 INFO:teuthology.orchestra.run.smithi069.stdout:/var/log/ceph/valgrind/mon.a.log: Leak_StillReachable 2023-10-30T20:54:47.813 INFO:teuthology.orchestra.run.smithi069.stdout:/var/log/ceph/valgrind/mon.c.log: Leak_StillReachable 2023-10-30T20:54:47.813 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind Leak_StillReachable 2023-10-30T20:54:47.814 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.a.log 2023-10-30T20:54:47.814 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind Leak_StillReachable 2023-10-30T20:54:47.814 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.c.log 2023-10-30T20:54:47.849 INFO:teuthology.orchestra.run.smithi183.stdout:/var/log/ceph/valgrind/mon.b.log: Leak_StillReachable 2023-10-30T20:54:47.849 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind Leak_StillReachable 2023-10-30T20:54:47.849 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.b.log 2023-10-30T20:54:47.850 INFO:tasks.ceph:Archiving crash dumps... 2023-10-30T20:54:47.851 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403/remote/smithi069/crash 2023-10-30T20:54:47.852 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T20:54:47.879 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403/remote/smithi183/crash 2023-10-30T20:54:47.880 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T20:54:47.915 INFO:tasks.ceph:Compressing logs... 2023-10-30T20:54:47.916 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T20:54:47.924 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T20:54:47.947 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89980.log 2023-10-30T20:54:47.947 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89866.log 2023-10-30T20:54:47.948 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.89980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91906.log 2023-10-30T20:54:47.977 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.89866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90958.log 2023-10-30T20:54:47.978 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91906.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89980.log.gz 2023-10-30T20:54:47.978 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158416.log 2023-10-30T20:54:47.978 INFO:teuthology.orchestra.run.smithi069.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.89866.log.gz 2023-10-30T20:54:47.978 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90958.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-30T20:54:47.978 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91800.log 2023-10-30T20:54:47.979 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mon.a.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90958.log.gz 2023-10-30T20:54:47.979 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86021.log 2023-10-30T20:54:47.979 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158416.log.gz 2023-10-30T20:54:47.979 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90916.log 2023-10-30T20:54:47.979 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89920.log 2023-10-30T20:54:47.979 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.86021.log: /var/log/ceph/ceph-client.admin.90916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127574.log 2023-10-30T20:54:47.980 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.89920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91650.log 2023-10-30T20:54:47.980 INFO:teuthology.orchestra.run.smithi069.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90916.log.gz 2023-10-30T20:54:47.980 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.127574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92180.log 2023-10-30T20:54:47.980 INFO:teuthology.orchestra.run.smithi069.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.89920.log.gz/var/log/ceph/ceph-client.admin.91650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92358.log 2023-10-30T20:54:47.980 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:54:47.980 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91256.log 2023-10-30T20:54:47.981 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92180.log: /var/log/ceph/ceph-client.admin.92358.log: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.86021.log.gz 2023-10-30T20:54:47.981 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91909.log 2023-10-30T20:54:47.981 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91256.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.91650.log.gz 2023-10-30T20:54:47.981 INFO:teuthology.orchestra.run.smithi069.stderr: 90.8% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91256.log.gz 2023-10-30T20:54:47.981 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.127574.log.gz 2023-10-30T20:54:47.981 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92268.log 2023-10-30T20:54:47.982 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87347.log 2023-10-30T20:54:47.982 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92268.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2023-10-30T20:54:47.982 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.87347.log: 88.2% 89.3% -- replaced with /var/log/ceph/ceph-client.admin.87347.log.gz 2023-10-30T20:54:47.982 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92268.log.gz 2023-10-30T20:54:47.982 INFO:teuthology.orchestra.run.smithi069.stderr: 90.9%gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2023-10-30T20:54:47.982 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92358.log.gz 2023-10-30T20:54:47.983 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2023-10-30T20:54:47.983 INFO:teuthology.orchestra.run.smithi069.stderr: 74.7%gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2023-10-30T20:54:47.983 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2023-10-30T20:54:47.983 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/osd.2.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2023-10-30T20:54:47.983 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2023-10-30T20:54:47.984 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2023-10-30T20:54:47.984 INFO:teuthology.orchestra.run.smithi069.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2023-10-30T20:54:47.984 INFO:teuthology.orchestra.run.smithi069.stderr: 63.9%/var/log/ceph/valgrind/mon.a.log: -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2023-10-30T20:54:47.984 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92171.log 2023-10-30T20:54:47.984 INFO:teuthology.orchestra.run.smithi069.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2023-10-30T20:54:47.984 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 91.1% -- replaced with /var/log/ceph/ceph-client.admin.91906.log.gz 2023-10-30T20:54:47.985 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.90340.log 2023-10-30T20:54:47.985 INFO:teuthology.orchestra.run.smithi069.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.91909.log.gz 2023-10-30T20:54:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92171.log: /var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92290.log 2023-10-30T20:54:47.985 INFO:teuthology.orchestra.run.smithi069.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2023-10-30T20:54:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91831.log 2023-10-30T20:54:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90332.log 2023-10-30T20:54:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91831.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.90340.log.gz 2023-10-30T20:54:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90957.log 2023-10-30T20:54:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157746.log 2023-10-30T20:54:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90957.log: /var/log/ceph/ceph-client.admin.90332.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.92171.log.gz 2023-10-30T20:54:47.986 INFO:teuthology.orchestra.run.smithi069.stderr: 87.4% 87.2% -- replaced with /var/log/ceph/ceph-client.admin.90332.log.gz 2023-10-30T20:54:47.987 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90957.log.gz 2023-10-30T20:54:47.987 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90652.log 2023-10-30T20:54:47.987 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158208.log 2023-10-30T20:54:47.987 INFO:teuthology.orchestra.run.smithi069.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.92290.log.gz 2023-10-30T20:54:47.987 INFO:teuthology.orchestra.run.smithi069.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92180.log.gz 2023-10-30T20:54:47.988 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158082.log 2023-10-30T20:54:47.988 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135448.log 2023-10-30T20:54:47.988 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.157746.log.gz 2023-10-30T20:54:47.988 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158208.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90652.log.gz 2023-10-30T20:54:47.988 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92013.log 2023-10-30T20:54:47.989 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.135448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86530.log 2023-10-30T20:54:47.989 INFO:teuthology.orchestra.run.smithi069.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.158082.log.gz 2023-10-30T20:54:47.989 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90657.log 2023-10-30T20:54:47.989 INFO:teuthology.orchestra.run.smithi069.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.158208.log.gz 2023-10-30T20:54:47.989 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.86530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158250.log 2023-10-30T20:54:47.989 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109322.log 2023-10-30T20:54:47.990 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86530.log.gz 2023-10-30T20:54:47.990 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8%/var/log/ceph/ceph-client.admin.158250.log: -- replaced with /var/log/ceph/ceph-client.admin.90657.log.gz 2023-10-30T20:54:47.990 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90653.log 2023-10-30T20:54:47.990 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.109322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91565.log 2023-10-30T20:54:47.990 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90653.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.158250.log.gz 2023-10-30T20:54:47.991 INFO:teuthology.orchestra.run.smithi069.stderr:gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109322.log.gz 2023-10-30T20:54:47.991 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.91058.log 2023-10-30T20:54:47.991 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90653.log.gz 2023-10-30T20:54:47.991 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91711.log 2023-10-30T20:54:47.991 INFO:teuthology.orchestra.run.smithi069.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.135448.log.gz 2023-10-30T20:54:47.991 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91565.log: /var/log/ceph/ceph-client.admin.91058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135447.log 2023-10-30T20:54:47.992 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92076.log 2023-10-30T20:54:47.992 INFO:teuthology.orchestra.run.smithi069.stderr: 90.8% 86.8% -- replaced with /var/log/ceph/ceph-client.admin.91058.log.gz 2023-10-30T20:54:47.992 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91831.log.gz 2023-10-30T20:54:47.992 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.135447.log: gzip -5 90.4% -- replaced with /var/log/ceph/ceph-client.admin.92013.log.gz 2023-10-30T20:54:47.992 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.158376.log 2023-10-30T20:54:47.992 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157788.log 2023-10-30T20:54:47.993 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92288.log 2023-10-30T20:54:47.993 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157788.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158376.log.gz 2023-10-30T20:54:47.993 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- 90.6% /var/log/ceph/ceph-client.admin.88276.log 2023-10-30T20:54:47.993 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.135447.log.gz 2023-10-30T20:54:47.993 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92288.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.157788.log.gz 2023-10-30T20:54:47.993 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90878.log 2023-10-30T20:54:47.994 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.88276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157451.log 2023-10-30T20:54:47.994 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90878.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.88276.log.gz 2023-10-30T20:54:47.994 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80214.log 2023-10-30T20:54:47.994 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90878.log.gz 2023-10-30T20:54:47.994 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157451.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T20:54:47.994 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.tmp-client.admin.80214.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80214.log.gz 2023-10-30T20:54:47.995 INFO:teuthology.orchestra.run.smithi069.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.157451.log.gz 2023-10-30T20:54:47.995 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157914.log 2023-10-30T20:54:47.995 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105957.log 2023-10-30T20:54:47.995 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157914.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.92288.log.gz 2023-10-30T20:54:47.995 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90662.log 2023-10-30T20:54:47.996 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.105957.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.157914.log.gz 2023-10-30T20:54:47.996 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90028.log 2023-10-30T20:54:47.996 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90662.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.92076.log.gz 2023-10-30T20:54:47.996 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90662.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90341.log 2023-10-30T20:54:47.996 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:54:47.996 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90164.log 2023-10-30T20:54:47.997 INFO:teuthology.orchestra.run.smithi069.stderr: 87.1%/var/log/ceph/ceph-client.admin.90341.log: -- replaced with /var/log/ceph/ceph-client.admin.90028.log.gz 2023-10-30T20:54:47.997 INFO:teuthology.orchestra.run.smithi069.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.105957.log.gz 2023-10-30T20:54:47.997 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90661.log 2023-10-30T20:54:47.997 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90164.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91688.log 2023-10-30T20:54:47.997 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90341.log.gz 2023-10-30T20:54:47.997 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90661.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90164.log.gz 2023-10-30T20:54:47.998 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92256.log 2023-10-30T20:54:47.998 INFO:teuthology.orchestra.run.smithi069.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90661.log.gz 2023-10-30T20:54:47.998 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105959.log 2023-10-30T20:54:47.998 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91688.log: /var/log/ceph/ceph-client.admin.92256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90343.log 2023-10-30T20:54:47.998 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.105959.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92256.log.gz 2023-10-30T20:54:47.999 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91608.log 2023-10-30T20:54:48.014 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-30T20:54:48.014 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2023-10-30T20:54:48.014 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2023-10-30T20:54:48.015 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2023-10-30T20:54:48.015 INFO:teuthology.orchestra.run.smithi183.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2023-10-30T20:54:48.015 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2023-10-30T20:54:48.015 INFO:teuthology.orchestra.run.smithi183.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2023-10-30T20:54:48.015 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2023-10-30T20:54:48.016 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/valgrind/osd.6.log: /var/log/ceph/valgrind/mon.b.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2023-10-30T20:54:48.016 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T20:54:48.016 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/valgrind/osd.4.log: 74.7% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2023-10-30T20:54:48.016 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86027.log 2023-10-30T20:54:48.016 INFO:teuthology.orchestra.run.smithi183.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2023-10-30T20:54:48.016 INFO:teuthology.orchestra.run.smithi183.stderr:gzip/var/log/ceph/ceph.audit.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.86107.log 2023-10-30T20:54:48.017 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.86027.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-30T20:54:48.017 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.86107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85987.log 2023-10-30T20:54:48.017 INFO:teuthology.orchestra.run.smithi183.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86027.log.gz 2023-10-30T20:54:48.017 INFO:teuthology.orchestra.run.smithi183.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.86107.log.gz/var/log/ceph/ceph-osd.4.log: 2023-10-30T20:54:48.017 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-30T20:54:48.017 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-30T20:54:48.018 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.85987.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85947.log 2023-10-30T20:54:48.018 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-30T20:54:48.018 INFO:teuthology.orchestra.run.smithi183.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.85987.log.gz 2023-10-30T20:54:48.018 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.85947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86198.log 2023-10-30T20:54:48.018 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86238.log 2023-10-30T20:54:48.018 INFO:teuthology.orchestra.run.smithi183.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.85947.log.gz 2023-10-30T20:54:48.019 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.86198.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T20:54:48.019 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.86238.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86198.log.gz 2023-10-30T20:54:48.019 INFO:teuthology.orchestra.run.smithi183.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-30T20:54:48.019 INFO:teuthology.orchestra.run.smithi183.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86238.log.gz 2023-10-30T20:54:48.019 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86147.log 2023-10-30T20:54:48.019 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T20:54:48.020 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86067.log 2023-10-30T20:54:48.020 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.86147.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.86147.log.gz 2023-10-30T20:54:48.020 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.86067.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T20:54:48.020 INFO:teuthology.orchestra.run.smithi183.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86067.log.gz 2023-10-30T20:54:48.020 INFO:teuthology.orchestra.run.smithi183.stderr: 94.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-30T20:54:48.034 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90343.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90343.log.gz 2023-10-30T20:54:48.035 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157956.log 2023-10-30T20:54:48.035 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91608.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.105959.log.gz 2023-10-30T20:54:48.035 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92094.log 2023-10-30T20:54:48.035 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157956.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.157956.log.gz 2023-10-30T20:54:48.035 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117764.log 2023-10-30T20:54:48.036 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92094.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.91688.log.gz 2023-10-30T20:54:48.036 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157872.log 2023-10-30T20:54:48.036 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.117764.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.117764.log.gz 2023-10-30T20:54:48.036 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-30T20:54:48.036 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157872.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.157872.log.gz 2023-10-30T20:54:48.037 INFO:teuthology.orchestra.run.smithi069.stderr: 95.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T20:54:48.037 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90655.log 2023-10-30T20:54:48.037 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92087.log 2023-10-30T20:54:48.037 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.90655.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.91608.log.gz 2023-10-30T20:54:48.037 INFO:teuthology.orchestra.run.smithi069.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90655.log.gz 2023-10-30T20:54:48.037 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91553.log 2023-10-30T20:54:48.038 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127573.log 2023-10-30T20:54:48.038 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91553.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.92094.log.gz 2023-10-30T20:54:48.038 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92331.log 2023-10-30T20:54:48.038 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.127573.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.127573.log.gz 2023-10-30T20:54:48.038 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157830.log 2023-10-30T20:54:48.039 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92331.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.92087.log.gz 2023-10-30T20:54:48.039 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86429.log 2023-10-30T20:54:48.039 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157830.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.157830.log.gz 2023-10-30T20:54:48.039 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157704.log 2023-10-30T20:54:48.039 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.86429.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86429.log.gz 2023-10-30T20:54:48.039 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109323.log 2023-10-30T20:54:48.040 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157704.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.157704.log.gz 2023-10-30T20:54:48.040 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92010.log 2023-10-30T20:54:48.040 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.109323.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.92331.log.gz 2023-10-30T20:54:48.040 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92363.log 2023-10-30T20:54:48.040 INFO:teuthology.orchestra.run.smithi069.stderr: 91.2%/var/log/ceph/ceph-client.admin.92010.log: -- replaced with /var/log/ceph/ceph-client.admin.109323.log.gz 2023-10-30T20:54:48.040 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91842.log 2023-10-30T20:54:48.041 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92363.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.92363.log.gz 2023-10-30T20:54:48.041 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90654.log 2023-10-30T20:54:48.041 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91842.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.92010.log.gz 2023-10-30T20:54:48.041 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91589.log 2023-10-30T20:54:48.041 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90654.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90654.log.gz 2023-10-30T20:54:48.041 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose 91.1% -- replaced with /var/log/ceph/ceph-client.admin.91842.log.gz 2023-10-30T20:54:48.042 INFO:teuthology.orchestra.run.smithi069.stderr: -- /var/log/ceph/ceph-client.admin.158166.log 2023-10-30T20:54:48.042 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91589.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-30T20:54:48.042 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158166.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.158166.log.gz 2023-10-30T20:54:48.042 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89645.log 2023-10-30T20:54:48.042 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.3.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.91589.log.gz 2023-10-30T20:54:48.043 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91214.log 2023-10-30T20:54:48.043 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.89645.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89645.log.gz 2023-10-30T20:54:48.043 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109321.log 2023-10-30T20:54:48.043 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91214.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.91214.log.gz 2023-10-30T20:54:48.043 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90334.log 2023-10-30T20:54:48.043 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.109321.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.109321.log.gz 2023-10-30T20:54:48.044 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92176.log 2023-10-30T20:54:48.044 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90334.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90334.log.gz 2023-10-30T20:54:48.044 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92382.log 2023-10-30T20:54:48.044 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92176.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.91553.log.gz 2023-10-30T20:54:48.044 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158334.log 2023-10-30T20:54:48.044 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92382.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.92382.log.gz 2023-10-30T20:54:48.045 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157493.log 2023-10-30T20:54:48.045 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158334.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.158334.log.gz 2023-10-30T20:54:48.045 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91078.log 2023-10-30T20:54:48.045 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157493.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.157493.log.gz 2023-10-30T20:54:48.045 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92381.log 2023-10-30T20:54:48.045 INFO:teuthology.orchestra.run.smithi069.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.91565.log.gz 2023-10-30T20:54:48.046 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105958.log 2023-10-30T20:54:48.046 INFO:teuthology.orchestra.run.smithi069.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91078.log.gz 2023-10-30T20:54:48.046 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158040.log 2023-10-30T20:54:48.046 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.105958.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.105958.log.gz 2023-10-30T20:54:48.046 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112546.log 2023-10-30T20:54:48.047 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158040.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158040.log.gz 2023-10-30T20:54:48.047 INFO:teuthology.orchestra.run.smithi069.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.92381.log.gz 2023-10-30T20:54:48.047 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91172.log 2023-10-30T20:54:48.047 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.112546.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-30T20:54:48.047 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91172.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.112546.log.gz 2023-10-30T20:54:48.055 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90337.log 2023-10-30T20:54:48.055 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91172.log.gz/var/log/ceph/ceph-mgr.y.log: 2023-10-30T20:54:48.055 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90345.log 2023-10-30T20:54:48.055 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90337.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90337.log.gz 2023-10-30T20:54:48.055 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158292.log 2023-10-30T20:54:48.056 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90345.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90345.log.gz 2023-10-30T20:54:48.056 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117765.log 2023-10-30T20:54:48.056 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158292.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.158292.log.gz 2023-10-30T20:54:48.056 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90072.log 2023-10-30T20:54:48.056 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.117765.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.117765.log.gz 2023-10-30T20:54:48.056 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157620.log 2023-10-30T20:54:48.057 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90072.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90072.log.gz 2023-10-30T20:54:48.057 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157577.log 2023-10-30T20:54:48.057 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157620.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.157620.log.gz 2023-10-30T20:54:48.057 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117763.log 2023-10-30T20:54:48.057 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157577.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.157577.log.gz 2023-10-30T20:54:48.058 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-30T20:54:48.058 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.117763.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.117763.log.gz 2023-10-30T20:54:48.058 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91630.log 2023-10-30T20:54:48.065 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127575.log 2023-10-30T20:54:48.074 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91630.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.92176.log.gz 2023-10-30T20:54:48.076 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86400.log 2023-10-30T20:54:48.079 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.127575.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.127575.log.gz 2023-10-30T20:54:48.089 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T20:54:48.090 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.86400.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.86400.log.gz 2023-10-30T20:54:48.102 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90118.log 2023-10-30T20:54:48.115 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.log: 93.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91719.log 2023-10-30T20:54:48.116 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90118.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90118.log.gz 2023-10-30T20:54:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-30T20:54:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91719.log: -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T20:54:48.130 INFO:teuthology.orchestra.run.smithi069.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.91630.log.gz 2023-10-30T20:54:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157998.log 2023-10-30T20:54:48.148 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91658.log 2023-10-30T20:54:48.149 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157998.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.157998.log.gz 2023-10-30T20:54:48.162 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158124.log 2023-10-30T20:54:48.171 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91658.log: 91.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135446.log 2023-10-30T20:54:48.180 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158124.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91658.log.gz 2023-10-30T20:54:48.181 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127577.log 2023-10-30T20:54:48.185 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.135446.log: -- replaced with /var/log/ceph/ceph-client.admin.158124.log.gz 2023-10-30T20:54:48.189 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157535.log 2023-10-30T20:54:48.195 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.127577.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.135446.log.gz 2023-10-30T20:54:48.205 INFO:teuthology.orchestra.run.smithi069.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.127577.log.gz 2023-10-30T20:54:48.206 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90998.log 2023-10-30T20:54:48.207 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157535.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.157535.log.gz 2023-10-30T20:54:48.227 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90682.log 2023-10-30T20:54:48.228 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90998.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90998.log.gz 2023-10-30T20:54:48.240 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90346.log 2023-10-30T20:54:48.241 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90682.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90682.log.gz 2023-10-30T20:54:48.250 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112543.log 2023-10-30T20:54:48.251 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90346.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90346.log.gz 2023-10-30T20:54:48.269 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127576.log 2023-10-30T20:54:48.272 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.112543.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.112543.log.gz 2023-10-30T20:54:48.278 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157662.log 2023-10-30T20:54:48.287 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.127576.log: 90.4%gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-30T20:54:48.294 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157662.log: 88.4% 94.2% -- replaced with /var/log/ceph/ceph-client.admin.127576.log.gz 2023-10-30T20:54:48.301 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mon.c.log: -- replaced with /var/log/ceph/ceph-client.admin.157662.log.gz 2023-10-30T20:54:48.304 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91719.log.gz 2023-10-30T20:54:48.377 INFO:teuthology.orchestra.run.smithi069.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-30T20:54:48.516 INFO:teuthology.orchestra.run.smithi069.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.admin.91800.log.gz 2023-10-30T20:54:49.411 INFO:teuthology.orchestra.run.smithi069.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.91711.log.gz 2023-10-30T20:54:49.826 INFO:teuthology.orchestra.run.smithi183.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-30T20:54:49.951 INFO:teuthology.orchestra.run.smithi069.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-30T20:54:50.990 INFO:teuthology.orchestra.run.smithi069.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-30T20:54:51.342 INFO:teuthology.orchestra.run.smithi183.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-30T20:54:51.404 INFO:teuthology.orchestra.run.smithi183.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-30T20:54:51.737 INFO:teuthology.orchestra.run.smithi069.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-30T20:54:51.782 INFO:teuthology.orchestra.run.smithi183.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-30T20:54:52.123 INFO:teuthology.orchestra.run.smithi183.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-30T20:54:52.125 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-10-30T20:54:52.125 INFO:teuthology.orchestra.run.smithi183.stderr:real 0m4.152s 2023-10-30T20:54:52.125 INFO:teuthology.orchestra.run.smithi183.stderr:user 0m16.039s 2023-10-30T20:54:52.126 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m0.549s 2023-10-30T20:54:52.130 INFO:teuthology.orchestra.run.smithi069.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-30T20:54:52.315 INFO:teuthology.orchestra.run.smithi069.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-30T20:54:52.494 INFO:teuthology.orchestra.run.smithi069.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-30T20:54:52.496 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-10-30T20:54:52.497 INFO:teuthology.orchestra.run.smithi069.stderr:real 0m4.561s 2023-10-30T20:54:52.497 INFO:teuthology.orchestra.run.smithi069.stderr:user 0m23.107s 2023-10-30T20:54:52.497 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m0.793s 2023-10-30T20:54:52.498 INFO:tasks.ceph:Archiving logs... 2023-10-30T20:54:52.498 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403/remote/smithi069/log 2023-10-30T20:54:52.500 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T20:54:55.298 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403/remote/smithi183/log 2023-10-30T20:54:55.299 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T20:54:57.378 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 368, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-10-30T20:54:57.379 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-30T20:54:57.390 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 368, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-10-30T20:54:57.390 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-30T20:54:57.391 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T20:54:57.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T20:54:57.487 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-30T20:54:57.495 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-30T20:54:57.496 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T20:54:57.498 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T20:54:57.512 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2023-10-30T20:54:57.515 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2023-10-30T20:54:57.517 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T20:54:57.517 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2023-10-30T20:54:57.517 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 +858us[ +858us] +/- 84ms 2023-10-30T20:54:57.517 INFO:teuthology.orchestra.run.smithi069.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -1272us[-1252us] +/- 74ms 2023-10-30T20:54:57.517 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 +592us[ +613us] +/- 89ms 2023-10-30T20:54:57.518 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T20:54:57.519 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T20:54:57.520 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2023-10-30T20:54:57.520 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 34 +880us[ +880us] +/- 84ms 2023-10-30T20:54:57.520 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -1263us[-1305us] +/- 74ms 2023-10-30T20:54:57.520 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +596us[ +596us] +/- 89ms 2023-10-30T20:54:57.520 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T20:54:57.521 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-30T20:54:57.530 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-30T20:54:57.531 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-30T20:54:57.541 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-30T20:54:57.657 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-30T20:54:57.758 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T20:54:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T20:54:57.792 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T20:54:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T20:54:57.825 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-30T20:54:57.834 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-30T20:54:57.845 INFO:teuthology.task.internal:Duration was 3089.298234 seconds 2023-10-30T20:54:57.845 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-30T20:54:57.855 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-30T20:54:57.856 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T20:54:57.858 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T20:54:57.899 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T20:54:57.909 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T20:54:58.191 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-30T20:54:58.191 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2023-10-30T20:54:58.192 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T20:54:58.212 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2023-10-30T20:54:58.213 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T20:54:58.259 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-30T20:54:58.259 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T20:54:58.263 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T20:54:58.327 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-30T20:54:58.328 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T20:54:58.331 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T20:54:58.464 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-30T20:54:58.474 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-30T20:54:58.475 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T20:54:58.502 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T20:54:58.530 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-30T20:54:58.540 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T20:54:58.546 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T20:54:58.568 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2023-10-30T20:54:58.595 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2023-10-30T20:54:58.635 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T20:54:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T20:54:58.651 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T20:54:58.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T20:54:58.691 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-30T20:54:58.701 INFO:teuthology.task.internal:Transferring archived files... 2023-10-30T20:54:58.702 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403/remote/smithi069 2023-10-30T20:54:58.702 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T20:54:58.743 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403/remote/smithi183 2023-10-30T20:54:58.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T20:54:58.782 INFO:teuthology.task.internal:Removing archive directory... 2023-10-30T20:54:58.782 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T20:54:58.785 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T20:54:58.830 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-30T20:54:58.841 INFO:teuthology.task.internal:Not uploading archives. 2023-10-30T20:54:58.842 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-30T20:54:58.854 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-30T20:54:58.855 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T20:54:58.857 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T20:54:58.869 INFO:teuthology.orchestra.run.smithi069.stdout: 265243 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 20:54 /home/ubuntu/cephtest 2023-10-30T20:54:58.884 INFO:teuthology.orchestra.run.smithi183.stdout: 265974 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 20:54 /home/ubuntu/cephtest 2023-10-30T20:54:58.885 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-30T20:54:58.896 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-30T20:54:58.908 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: saw valgrind issues 2023-10-30T20:54:58.908 INFO:teuthology.nuke:Checking targets against current locks 2023-10-30T20:54:58.934 DEBUG:teuthology.nuke:shortname: smithi069 2023-10-30T20:54:58.935 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T20:54:58.963 DEBUG:teuthology.nuke:shortname: smithi183 2023-10-30T20:54:58.963 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T20:54:58.969 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 19:55:06.895013', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPYOadPJGZ6Dgmh4cUhlkNoyy6TxoaN3szWWY15zj2N5X3SpqTC5b49HSfaNZti1N2YwzFEGieaf0YsyG3qHro='} 2023-10-30T20:54:58.985 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2023-10-30_17:23:32-rados:verify-main-distro-default-smithi/7441403', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 19:55:06.893325', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBDn5rofjGLsweyWrhB4yDSdrLYw67ggLPiGlfBteC8yQzlu+mrd5uMp5UD/byLTDpd2QtVo9ZCEH6z8FMVeIQk='} 2023-10-30T20:54:59.016 INFO:teuthology.orchestra.console.smithi069:Power off 2023-10-30T20:54:59.017 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T20:54:59.044 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2023-10-30T20:54:59.045 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T20:54:59.061 INFO:teuthology.orchestra.console.smithi183:Power off 2023-10-30T20:54:59.062 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T20:54:59.074 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-10-30T20:54:59.087 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2023-10-30T20:54:59.088 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T20:54:59.115 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2023-10-30T20:55:03.076 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T20:55:03.116 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T20:55:03.185 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-10-30T20:55:03.224 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2023-10-30T20:55:07.188 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T20:55:07.225 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T20:55:07.300 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2023-10-30T20:55:07.336 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2023-10-30T20:55:07.402 INFO:teuthology.orchestra.console.smithi069:Power off completed 2023-10-30T20:55:07.437 INFO:teuthology.orchestra.console.smithi183:Power off completed 2023-10-30T20:55:07.542 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} duration: 3089.2982335090637 failure_reason: saw valgrind issues flavor: default owner: scheduled_amathuri@teuthology status: fail success: false 2023-10-30T20:55:07.542 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T20:55:07.618 INFO:teuthology.run:FAIL